Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 163.660678ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:11:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 163.685697ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 176.026946ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:11:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 176.051242ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X....XX........X...................................] duration: 5.98729501s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:11:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.987325678s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 170.207023ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:10:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 170.231741ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X....XX........X..................................] duration: 5.963786645s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:10:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.963811222s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 159.478565ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:09:42 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 159.506928ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X....XX........X.................................] duration: 6.031649819s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:09:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.031674326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X........X....XX........X................................] duration: 15.963665231s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:08:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 15.963691151s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 135.109482ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:08:32 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 135.141743ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X........X....XX........X...............................] duration: 6.096732798s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:07:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.096750061s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 221.608551ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:07:42 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 221.640882ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X........X....XX........X..............................] duration: 18.144486545s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 18.144509958s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 141.335667ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:07:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 141.360154ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X........X....XX........X.............................] duration: 5.860551267s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:06:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.860575855s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 215.082955ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:05:59 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 215.112822ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X......X........X....XX........X............................] duration: 6.670913918s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:05:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.670931792s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 135.069136ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:05:28 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 135.138677ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 161.843108ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:05:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 161.869597ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X......X........X....XX........X...........................] duration: 6.000956794s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:04:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.000980508s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 158.34167ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:04:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 158.370895ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 198.107522ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:04:25 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 198.128281ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X......X........X....XX........X..........................] duration: 5.943242987s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:04:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.943271872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X......X........X....XX........X.........................] duration: 7.288879213s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:03:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 7.288903138s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 181.12299ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:03:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 181.137929ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001217847s trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:03:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001235021s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X......X........X....XX........X........................] duration: 5.895149589s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:02:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.895185597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X......X........X....XX........X.......................] duration: 7.516285193s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:01:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 7.516313145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X......X........X....XX........X......................] duration: 6.311607818s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:01:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.311635801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X......X........X....XX........X.....................] duration: 22.636002353s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 22:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 22.636019956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X......X........X....XX........X....................] duration: 6.038284623s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:59:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.038300974s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X......X........X....XX........X...................] duration: 6.933242447s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:59:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.933276332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X......X........X....XX........X..................] duration: 9.271084586s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:58:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 9.271111617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..........X......X........X....XX........X.................] duration: 5.941133821s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:57:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.941163427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..........X......X........X....XX........X................] duration: 13.149248163s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:56:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 13.149273451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..........X......X........X....XX........X...............] duration: 7.417336785s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:55:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 7.417366029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..........X......X........X....XX........X..............] duration: 7.818339405s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:54:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 7.818370864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..........X......X........X....XX........X.............] duration: 6.337257084s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:54:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.337287032s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..........X......X........X....XX........X............] duration: 10.914496729s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:53:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 10.914524451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..........X......X........X....XX........X...........] duration: 6.552798212s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.552827498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..........X......X........X....XX........X..........] duration: 6.091506845s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.091525019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..........X......X........X....XX........X.........] duration: 19.940997623s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 19.941026559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X..........X......X........X....XX........X........] duration: 5.900820374s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.900847166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X..........X......X........X....XX........X.......] duration: 21.948758481s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 21.948817616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X..........X......X........X....XX........X......] duration: 23.635454365s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 23.635480967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X..........X......X........X....XX........X.....] duration: 16.106894602s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 16.106923638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X..........X......X........X....XX........X....] duration: 15.834723881s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:46:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 15.834750872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X..........X......X........X....XX........X...] duration: 6.453565005s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:45:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.453585465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X..........X......X........X....XX........X..] duration: 17.241053389s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 17.241081363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X..........X......X........X....XX........X.] duration: 25.30439053s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:44:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 25.304417352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X..........X......X........X....XX........X] duration: 30.002978533s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:44:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 30.003002949s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)