Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..........X......X........X....XX........X............] duration: 10.914496729s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:53:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 10.914524451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..........X......X........X....XX........X...........] duration: 6.552798212s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.552827498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..........X......X........X....XX........X..........] duration: 6.091506845s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.091525019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..........X......X........X....XX........X.........] duration: 19.940997623s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 19.941026559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X..........X......X........X....XX........X........] duration: 5.900820374s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 5.900847166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X..........X......X........X....XX........X.......] duration: 21.948758481s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 21.948817616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X..........X......X........X....XX........X......] duration: 23.635454365s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 23.635480967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X..........X......X........X....XX........X.....] duration: 16.106894602s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 16.106923638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X..........X......X........X....XX........X....] duration: 15.834723881s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:46:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 15.834750872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X..........X......X........X....XX........X...] duration: 6.453565005s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:45:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 6.453585465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X..........X......X........X....XX........X..] duration: 17.241053389s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 17.241081363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X..........X......X........X....XX........X.] duration: 25.30439053s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:44:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 25.304417352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X..........X......X........X....XX........X] duration: 30.002978533s trace: https://jaeger.dev.storj.io/trace/6d7f87848bf84981 FAIL (2025-12-27 21:44:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:43:52.497517497 (534h14m52.368600532s) [upload=468546]: Starting upload Dec 27 21:43:52.497594264 (534h14m52.368677299s) [upload=468546]: Done starting upload Dec 27 21:43:52.497696719 (534h14m52.368779784s) [upload=468546]: Uploading segments... Dec 27 21:43:52.497776301 (534h14m52.368859346s) [upload=468546]: Got next segment. Inline: false Dec 27 21:43:52.497935193 (534h14m52.369018228s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:43:52.497942097 (534h14m52.369025142s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:43:52.739151340 (534h14m52.610234386s) [seg_pos={0 0}, upload=468546]: Begin upload segment... Dec 27 21:43:52.739193811 (534h14m52.610276846s) [seg_pos={0 0}, upload=468546]: Done begin upload segment. Dec 27 21:43:52.739221193 (534h14m52.610304239s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739279154 (534h14m52.610362199s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.739726075 (534h14m52.61080912s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740041696 (534h14m52.611124731s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740315939 (534h14m52.611398974s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740693187 (534h14m52.611776222s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.740984371 (534h14m52.612067406s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741270255 (534h14m52.6123533s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.741619410 (534h14m52.612702445s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.743069442 (534h14m52.614152477s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.744335913 (534h14m52.615418958s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.745396944 (534h14m52.616479979s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.746785217 (534h14m52.617868262s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.749125003 (534h14m52.620208048s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.751777383 (534h14m52.622860428s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:52.752945557 (534h14m52.624028592s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Uploading piece... Dec 27 21:43:54.453502955 (534h14m54.32458599s) [upload=468546]: Next returned nil segment Dec 27 21:43:54.453520719 (534h14m54.324603785s) [upload=468546]: Waiting for error group managing segments... Dec 27 21:43:54.737745103 (534h14m54.608828138s) [noise=true , address=145.79.227.9:4365 , piece_id=QXB22AOEL3LCXSVJYLQ6RNX5WMOIRFYH4HNBP4JIEKBPBKOIIC5Q, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.748606204 (534h14m54.619689249s) [noise=true , address=145.79.227.3:4151 , piece_id=W23EYYDPLQB7PTV5XNUA7IOMPA57FS7ELLW6ZQQSLQEXAGZ5UPZQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.750761748 (534h14m54.621844783s) [noise=true , address=145.79.227.23:4547 , piece_id=BNZYII3DL6UWBUO4OV6TPCMUHLWNK7Q7D7FVVHO5XEUILHZIR2RA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.759067574 (534h14m54.630150609s) [noise=true , address=145.79.227.42:4121 , piece_id=ETON6EIOUSI3EGXUS223MHFHQZ2HXPDKQ7O5MQ4MKO4IGIIBLOFQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760024016 (534h14m54.631107061s) [noise=true , address=145.79.227.46:4602 , piece_id=PEY4HIHR3A2WT4HGBXRSASGZIS6BEON6S63MKD2FJCD54JVIHGGA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.760280494 (534h14m54.631363529s) [noise=true , address=145.79.227.10:4402 , piece_id=T36PV56QGANX6OVYERWA6SIDBYBDWDJMXM5ZXAMBWU3OONHU5FXA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.768196827 (534h14m54.639279873s) [noise=true , address=145.79.227.2:4113 , piece_id=V3IBPHBBLE5PWU3XAY5QJTJU2KP5JJH7OL33MJ4KVOY2JBNA7UDQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.779648604 (534h14m54.650731649s) [noise=true , address=145.79.227.38:4979 , piece_id=WFADFZYEWP5DJE5E5IG7AT4FTPOD6KCLVXE3NOXCZK7YTXMP34HQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.865859695 (534h14m54.73694275s) [noise=true , address=145.79.227.37:4566 , piece_id=MVYH3P6VNHQWBA6QUWS73TXBVQOSITHFJTG5DSKQSEN3RVIULKCA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.940797329 (534h14m54.811880364s) [noise=true , address=145.79.227.16:4909 , piece_id=ZH4QL3BVUKCS3UONG2VKVJEIIKJH5IOW2JUF4G6AVOFKOLPGITNA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.956726331 (534h14m54.827809376s) [noise=true , address=45.91.116.12:4228 , piece_id=ZGT373AHKGQVGXZFHX6XP7DGWZYHZFT5IW75IMHNQCAQ7ELUD2QQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995610644 (534h14m54.866693679s) [noise=true , address=77.90.31.12:4589 , piece_id=U54JQDFLGCI4IXBYYPXSAZMNMKJGB5O3FAF7QJB7U7QEMOAMBSYA, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995712498 (534h14m54.866795533s) [noise=true , address=45.91.116.9:4263 , piece_id=K45K4G6FP6R2KXLNBIBUHVSHZCVTZTKONPIWWI2XA6U7BINIGDKQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: <nil> Dec 27 21:43:54.995737234 (534h14m54.866820279s) [seg_pos={0 0}, upload=468546]: Segment reached optimal threshold of 13 pieces. Dec 27 21:43:54.995910985 (534h14m54.86699402s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995917577 (534h14m54.867000612s) [noise=true , address=45.91.116.7:4010 , piece_id=7BDNQUHIAZEIJX4X434QMA34R7D5FY2GBHC5GTUHX3GI7KDO3YPQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995928549 (534h14m54.867011584s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995934510 (534h14m54.867017545s) [noise=true , address=77.90.31.21:4880 , piece_id=5O2LGKIV2FYR6CRSN7IIVJOGZSTNQ7BBZJ6ED25ZX2JZFYGZZEGQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.995992130 (534h14m54.867075175s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 21:43:54.995996909 (534h14m54.867079954s) [noise=true , address=45.91.116.6:4975 , piece_id=XCPIMIULUMC75YMAFXUZGG2QIYNQGBYDA7KBDTADWZZ5JJHOLOQQ, seg_pos={0 0}, upload=468546]: Long tail cancellation due to optimal threshold Dec 27 21:43:54.996009493 (534h14m54.867092528s) [seg_pos={0 0}, upload=468546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:43:54.996068946 (534h14m54.867151991s) [upload=468546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:43:54.996073274 (534h14m54.867156309s) [upload=468546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:43:55.277286363 (534h14m55.148369408s) [upload=468546]: Done uploading segments... Dec 27 21:43:55.277308154 (534h14m55.148391189s) [upload=468546]: Upload finished. err: <nil> Duration: 30.003002949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X..........X......X........X....XX........] duration: 6.580318006s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:42:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 6.580356309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X..........X......X........X....XX.......] duration: 6.070956181s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:42:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 6.070989625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X..........X......X........X....XX......] duration: 20.027660244s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:41:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 20.027677196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....................X..........X......X........X....XX.....] duration: 25.129558369s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:40:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 25.129583306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....................X..........X......X........X....XX....] duration: 7.485614097s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 7.485643613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....................X..........X......X........X....XX...] duration: 6.046973364s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:39:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 6.046996638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X....................X..........X......X........X....XX..] duration: 14.529922942s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:38:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 14.529950374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX..X....................X..........X......X........X....XX.] duration: 6.056519397s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:37:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 6.056550526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX..X....................X..........X......X........X....XX] duration: 30.002391744s trace: https://jaeger.dev.storj.io/trace/42625c5aef7cf4e9 FAIL (2025-12-27 21:37:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ) context deadline exceeded ======LOGS====== Dec 27 21:36:31.082007007 (534h7m30.953090042s) [upload=468437]: Starting upload Dec 27 21:36:31.082064717 (534h7m30.953147752s) [upload=468437]: Done starting upload Dec 27 21:36:31.082203832 (534h7m30.953286867s) [upload=468437]: Uploading segments... Dec 27 21:36:31.082266511 (534h7m30.953349546s) [upload=468437]: Got next segment. Inline: false Dec 27 21:36:31.082309653 (534h7m30.953392718s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:36:31.082315965 (534h7m30.953399s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:36:31.324189314 (534h7m31.195272369s) [seg_pos={0 0}, upload=468437]: Begin upload segment... Dec 27 21:36:31.324255060 (534h7m31.195338105s) [seg_pos={0 0}, upload=468437]: Done begin upload segment. Dec 27 21:36:31.324289625 (534h7m31.195372661s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324359268 (534h7m31.195442313s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.324837289 (534h7m31.195920334s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325219486 (534h7m31.196302521s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.325553121 (534h7m31.196636156s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.326884386 (534h7m31.197967441s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.327285871 (534h7m31.198368916s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328654276 (534h7m31.199737321s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.328743446 (534h7m31.199826481s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330105881 (534h7m31.201188926s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.330135507 (534h7m31.201218552s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.331171610 (534h7m31.202254645s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.332347388 (534h7m31.203430433s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.333295414 (534h7m31.204378449s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.334518914 (534h7m31.205601959s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:31.336018810 (534h7m31.207101855s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Uploading piece... Dec 27 21:36:33.033543137 (534h7m32.904626182s) [upload=468437]: Next returned nil segment Dec 27 21:36:33.033566962 (534h7m32.904650007s) [upload=468437]: Waiting for error group managing segments... Dec 27 21:36:33.317135808 (534h7m33.188218843s) [noise=true , address=145.79.227.48:4302 , piece_id=XD6PFRGU5LMQTM43FHOCANR33GQAXC3B3NOYEQ4BQPTCDRTZBANA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.326105558 (534h7m33.197188603s) [noise=true , address=145.79.227.21:4474 , piece_id=EQABTEHVN4QE3QRLBKUVD22VLD774VN45N6OEEIVWW5IAR35OOLQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.330376139 (534h7m33.201459174s) [noise=true , address=145.79.227.12:4765 , piece_id=W7W7ST7RABXOUXE6Y74Z36O6AQ5KFWPLAOTNUGR2CK7ZZSE5JQCQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.333349471 (534h7m33.204432506s) [noise=true , address=145.79.227.37:4567 , piece_id=7X3CN75IPYNLGMDU4YSSI5A7Q7FBTIRUTJYMBRPHGNF7PHTGZUTQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.338509867 (534h7m33.209592913s) [noise=true , address=145.79.227.7:4295 , piece_id=2XJFVVOKGWWZ5WR6X4NE643YNLGH3BI5HH3P6QUJP5B5H3CNACQA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.347855233 (534h7m33.218938298s) [noise=true , address=145.79.227.5:4221 , piece_id=YUEAARDENXSTP4AI4N2QLBNVC53HCYRSTDQUNRSNH3FKXUOYUSMQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.352654711 (534h7m33.223737746s) [noise=true , address=145.79.227.9:4367 , piece_id=XQY5U54OAWNJDJABD7TUHG3LCROL5BVLXL5JPIXWG554U6C4NSGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.357715539 (534h7m33.228798574s) [noise=true , address=145.79.227.14:4837 , piece_id=WORNBXAAMWBE7J2AEFHCUA7B54OM5PY44TJHTUN3LTX6MPYCDJRQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.358935462 (534h7m33.230018507s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7BWI7DIOXLDNWM47KBYPD6MLRPBN4XJJCG7366JUFUNMTQNPCEQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.372480744 (534h7m33.243563779s) [noise=true , address=145.79.227.36:4519 , piece_id=RBTKLCK6A4MYTYY37BN3XBSV77HTQI5P5344LKSPRARFP2WNF4GA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.530812235 (534h7m33.40189528s) [noise=true , address=45.91.116.5:4297 , piece_id=567UXD55OE4OA5PRWNNAFPPJ245GRDOZ6X3KAGPZEVLVJYQZBMVA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.559045767 (534h7m33.430128812s) [noise=true , address=45.91.116.10:4083 , piece_id=VUAJ5MBAMSLKM334SIEJBGGZS6DBRMH3F5IEU4L3SIVCPTXK5HJQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565680561 (534h7m33.436763606s) [noise=true , address=45.91.116.7:4010 , piece_id=HFU255X3X6P4PFLASUQP7YHOBJKTAXKIILAZ5HXBTOOK4E462L5A, seg_pos={0 0}, upload=468437]: Done uploading piece. err: <nil> Dec 27 21:36:33.565703154 (534h7m33.436786189s) [seg_pos={0 0}, upload=468437]: Segment reached optimal threshold of 13 pieces. Dec 27 21:36:33.566008044 (534h7m33.437091089s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566017022 (534h7m33.437100057s) [noise=true , address=45.91.116.12:4226 , piece_id=QNYOD4CVPCETGMZIMDD7MZ6CZN5UQNHXIVHYH243OWPU3DIFWLZA, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566299790 (534h7m33.437382835s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566316311 (534h7m33.437399356s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:36:33.566328243 (534h7m33.437411279s) [noise=true , address=45.91.116.13:4155 , piece_id=ZKC4HOMLUQAQG4UHCUUSAUXB335DUDT74BBDRQ3T4TEQS3AR46HQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566358461 (534h7m33.437441496s) [noise=true , address=77.90.31.11:4554 , piece_id=VF2INNZXXWTIHXDP2PHPPKZDSMLJ3YWHUVYO4OBDTDEE2KUJ5EGQ, seg_pos={0 0}, upload=468437]: Long tail cancellation due to optimal threshold Dec 27 21:36:33.566379271 (534h7m33.437462306s) [seg_pos={0 0}, upload=468437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:36:33.566456408 (534h7m33.437539443s) [upload=468437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:36:33.566462840 (534h7m33.437545875s) [upload=468437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:36:33.854656462 (534h7m33.725739498s) [upload=468437]: Done uploading segments... Dec 27 21:36:33.854689846 (534h7m33.725772891s) [upload=468437]: Upload finished. err: <nil> Duration: 30.002409057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX..X....................X..........X......X........X....X] duration: 30.003245029s trace: https://jaeger.dev.storj.io/trace/56de98846b7ef456 FAIL (2025-12-27 21:36:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 21:35:46.090576776 (534h6m45.961659821s) [upload=468428]: Starting upload Dec 27 21:35:46.090631640 (534h6m45.961714675s) [upload=468428]: Done starting upload Dec 27 21:35:46.090816923 (534h6m45.961899958s) [upload=468428]: Uploading segments... Dec 27 21:35:46.090968712 (534h6m45.962051747s) [upload=468428]: Got next segment. Inline: false Dec 27 21:35:46.091020019 (534h6m45.962103064s) [upload=468428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:35:46.091024939 (534h6m45.962107975s) [upload=468428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:35:46.333451792 (534h6m46.204534827s) [seg_pos={0 0}, upload=468428]: Begin upload segment... Dec 27 21:35:46.333494373 (534h6m46.204577408s) [seg_pos={0 0}, upload=468428]: Done begin upload segment. Dec 27 21:35:46.333520944 (534h6m46.204603979s) [noise=true , address=145.79.227.45:4229 , piece_id=UME2UKJQIQJNZTKWBSUCMIRDFK6DL2HJIQZXGRCRD7HWTEBK63TA, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.333570438 (534h6m46.204653474s) [noise=true , address=145.79.227.52:4445 , piece_id=CMGVWO6QNJFOD3JCK3EEB2ZQAJMKJ2OPQQRYJH2WUUDEUB2UEK2A, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.334303404 (534h6m46.205386439s) [noise=true , address=145.79.227.46:4603 , piece_id=EU4IDN7VMIZCYBLLAPKVQTQG4LFVZPDQWI7XGZH6GQZXBIUEMLOA, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.334597965 (534h6m46.205681s) [noise=true , address=77.90.31.14:4663 , piece_id=AI47G2Q575LOQ7GGN2K6DCIZDLTCWEE3AYDLUBHAYMANEDW3ATGQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.334922182 (534h6m46.206005227s) [noise=true , address=145.79.227.48:4302 , piece_id=P53LAORWAQUW3TMFQUM3ZBET3E3RXH2AYLVC5VNZ4RPAA6AKEWAQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.335184031 (534h6m46.206267066s) [noise=true , address=145.79.227.7:4293 , piece_id=564JRFX3MOXLCGPISLVQEBVRNV43GH2AHYHHWNBPTX4FOD6PNDWQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.335252461 (534h6m46.206335506s) [noise=true , address=45.91.116.13:4155 , piece_id=P3AF345W7HVP5WKBWSUVCVXBTSDYZ23PYVYJ6SN6E2JN6Y5UJQHQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.335447432 (534h6m46.206530467s) [noise=true , address=145.79.227.20:4438 , piece_id=5D3XA5MJFGVTMNDLKLMTY5BYTQ2OGEPCWM72ZI53AIH35QXVD6NQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.335610973 (534h6m46.206694008s) [noise=true , address=45.91.116.11:4119 , piece_id=6LR4MXSZV45NB6U4GHZ4WLTGBQZTGSV5Y7KTZNMKDF4X5YIK7KWQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.335726504 (534h6m46.206809539s) [noise=true , address=45.91.116.14:4192 , piece_id=BQYHGYAOCFLTYHAAJ5TSD5KDCYQLTRFX72QXXZSXICYHKGEU7WNA, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.336000575 (534h6m46.20708361s) [noise=true , address=145.79.227.4:4186 , piece_id=RMHOUYOA6QYFCFPR7XN2OSOZ3ZTLPP6UGH5HSBVDPJKQPJBTTCMQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.336031474 (534h6m46.207114509s) [noise=true , address=145.79.227.9:4367 , piece_id=HM4BKGVLLADUKYOAVSO57GXKQ52KTYQBF2BXUXOY3URJNJ72BUSQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.336295046 (534h6m46.207378081s) [noise=true , address=45.91.116.8:4047 , piece_id=CVMR5UPU26DK2AHSEJX56G2AQTSAZ7GTTJBLLQSZARCM5KBV3PTQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.336381961 (534h6m46.207465006s) [noise=true , address=145.79.227.12:4765 , piece_id=YMCDGTN5YMXFBFRV5V7WCV3YOZFDNME5XVU7AZX46PRZIPQZFPRQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.336587192 (534h6m46.207670227s) [noise=true , address=77.90.31.21:4877 , piece_id=VYIATBHJ35DXH3N3W36WC6N3BIZWGO3CLA275C4HC2LL6DXDXPCQ, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:46.336732970 (534h6m46.207816005s) [noise=true , address=145.79.227.40:4049 , piece_id=BVXS2H25RRSSAZFYCHYFZTGLMA4DJNQSZAJEBLIEOF6Z2RQ5IMJA, seg_pos={0 0}, upload=468428]: Uploading piece... Dec 27 21:35:48.051243452 (534h6m47.922326487s) [upload=468428]: Next returned nil segment Dec 27 21:35:48.051273720 (534h6m47.922356765s) [upload=468428]: Waiting for error group managing segments... Dec 27 21:35:48.327897571 (534h6m48.198980616s) [noise=true , address=145.79.227.12:4765 , piece_id=YMCDGTN5YMXFBFRV5V7WCV3YOZFDNME5XVU7AZX46PRZIPQZFPRQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.337889878 (534h6m48.208972913s) [noise=true , address=145.79.227.9:4367 , piece_id=HM4BKGVLLADUKYOAVSO57GXKQ52KTYQBF2BXUXOY3URJNJ72BUSQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.338288817 (534h6m48.209371852s) [noise=true , address=145.79.227.48:4302 , piece_id=P53LAORWAQUW3TMFQUM3ZBET3E3RXH2AYLVC5VNZ4RPAA6AKEWAQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.341347642 (534h6m48.212430677s) [noise=true , address=145.79.227.7:4293 , piece_id=564JRFX3MOXLCGPISLVQEBVRNV43GH2AHYHHWNBPTX4FOD6PNDWQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.344374385 (534h6m48.21545742s) [noise=true , address=145.79.227.45:4229 , piece_id=UME2UKJQIQJNZTKWBSUCMIRDFK6DL2HJIQZXGRCRD7HWTEBK63TA, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.344620895 (534h6m48.21570394s) [noise=true , address=145.79.227.4:4186 , piece_id=RMHOUYOA6QYFCFPR7XN2OSOZ3ZTLPP6UGH5HSBVDPJKQPJBTTCMQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.345902956 (534h6m48.216985991s) [noise=true , address=145.79.227.46:4603 , piece_id=EU4IDN7VMIZCYBLLAPKVQTQG4LFVZPDQWI7XGZH6GQZXBIUEMLOA, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.357541668 (534h6m48.228624713s) [noise=true , address=145.79.227.20:4438 , piece_id=5D3XA5MJFGVTMNDLKLMTY5BYTQ2OGEPCWM72ZI53AIH35QXVD6NQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.375851403 (534h6m48.246934438s) [noise=true , address=145.79.227.52:4445 , piece_id=CMGVWO6QNJFOD3JCK3EEB2ZQAJMKJ2OPQQRYJH2WUUDEUB2UEK2A, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.527275281 (534h6m48.398358326s) [noise=true , address=77.90.31.14:4663 , piece_id=AI47G2Q575LOQ7GGN2K6DCIZDLTCWEE3AYDLUBHAYMANEDW3ATGQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.544161565 (534h6m48.4152446s) [noise=true , address=77.90.31.21:4877 , piece_id=VYIATBHJ35DXH3N3W36WC6N3BIZWGO3CLA275C4HC2LL6DXDXPCQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.563990483 (534h6m48.435073529s) [noise=true , address=45.91.116.8:4047 , piece_id=CVMR5UPU26DK2AHSEJX56G2AQTSAZ7GTTJBLLQSZARCM5KBV3PTQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.587972920 (534h6m48.459055955s) [noise=true , address=45.91.116.11:4119 , piece_id=6LR4MXSZV45NB6U4GHZ4WLTGBQZTGSV5Y7KTZNMKDF4X5YIK7KWQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: <nil> Dec 27 21:35:48.587991705 (534h6m48.459074751s) [seg_pos={0 0}, upload=468428]: Segment reached optimal threshold of 13 pieces. Dec 27 21:35:48.588165587 (534h6m48.459248632s) [noise=true , address=45.91.116.14:4192 , piece_id=BQYHGYAOCFLTYHAAJ5TSD5KDCYQLTRFX72QXXZSXICYHKGEU7WNA, seg_pos={0 0}, upload=468428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 21:35:48.588172710 (534h6m48.459255755s) [noise=true , address=45.91.116.14:4192 , piece_id=BQYHGYAOCFLTYHAAJ5TSD5KDCYQLTRFX72QXXZSXICYHKGEU7WNA, seg_pos={0 0}, upload=468428]: Long tail cancellation due to optimal threshold Dec 27 21:35:48.588266048 (534h6m48.459349083s) [noise=true , address=145.79.227.40:4049 , piece_id=BVXS2H25RRSSAZFYCHYFZTGLMA4DJNQSZAJEBLIEOF6Z2RQ5IMJA, seg_pos={0 0}, upload=468428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 27 21:35:48.588272029 (534h6m48.459355074s) [noise=true , address=145.79.227.40:4049 , piece_id=BVXS2H25RRSSAZFYCHYFZTGLMA4DJNQSZAJEBLIEOF6Z2RQ5IMJA, seg_pos={0 0}, upload=468428]: Long tail cancellation due to optimal threshold Dec 27 21:35:48.588386979 (534h6m48.459470014s) [noise=true , address=45.91.116.13:4155 , piece_id=P3AF345W7HVP5WKBWSUVCVXBTSDYZ23PYVYJ6SN6E2JN6Y5UJQHQ, seg_pos={0 0}, upload=468428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 21:35:48.588392539 (534h6m48.459475584s) [noise=true , address=45.91.116.13:4155 , piece_id=P3AF345W7HVP5WKBWSUVCVXBTSDYZ23PYVYJ6SN6E2JN6Y5UJQHQ, seg_pos={0 0}, upload=468428]: Long tail cancellation due to optimal threshold Dec 27 21:35:48.588408138 (534h6m48.459491174s) [seg_pos={0 0}, upload=468428]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:35:48.588474996 (534h6m48.459558031s) [upload=468428]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:35:48.588479594 (534h6m48.45956263s) [upload=468428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:35:48.869350212 (534h6m48.740433267s) [upload=468428]: Done uploading segments... Dec 27 21:35:48.869373446 (534h6m48.740456491s) [upload=468428]: Upload finished. err: <nil> Duration: 30.003269866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..XX..X....................X..........X......X........X....] duration: 6.014891349s trace: https://jaeger.dev.storj.io/trace/39d2c7777fcb15f8 FAIL (2025-12-27 21:34:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ) context deadline exceeded ======LOGS====== (534h2m47.392842023s) [upload=468370]: Uploading segments... Dec 27 21:31:47.521936445 (534h2m47.39301948s) [upload=468370]: Got next segment. Inline: false Dec 27 21:31:47.521974357 (534h2m47.393057393s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:31:47.521979097 (534h2m47.393062132s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:31:47.767045857 (534h2m47.638128902s) [seg_pos={0 0}, upload=468370]: Begin upload segment... Dec 27 21:31:47.767092054 (534h2m47.638175089s) [seg_pos={0 0}, upload=468370]: Done begin upload segment. Dec 27 21:31:47.767139495 (534h2m47.63822253s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767159122 (534h2m47.638242167s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767660247 (534h2m47.638743282s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767861990 (534h2m47.638945025s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768148817 (534h2m47.639231852s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768483103 (534h2m47.639566148s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768792382 (534h2m47.639875417s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769038832 (534h2m47.640121877s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769312453 (534h2m47.640395488s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769559002 (534h2m47.640642037s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769877889 (534h2m47.640960924s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770064996 (534h2m47.641148031s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770233427 (534h2m47.641316462s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771289227 (534h2m47.642372272s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771584158 (534h2m47.642667193s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771872227 (534h2m47.642955272s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:49.490998508 (534h2m49.362081543s) [upload=468370]: Next returned nil segment Dec 27 21:31:49.491016102 (534h2m49.362099137s) [upload=468370]: Waiting for error group managing segments... Dec 27 21:31:49.770955245 (534h2m49.64203829s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.771058903 (534h2m49.642141938s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.777122399 (534h2m49.648205434s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.790164273 (534h2m49.661247308s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.823247729 (534h2m49.694330764s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.826146289 (534h2m49.697229334s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.847994842 (534h2m49.719077887s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.996799172 (534h2m49.867882207s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047476210 (534h2m49.918559256s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047674247 (534h2m49.918757282s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.048428333 (534h2m49.919511378s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.067288637 (534h2m49.938371682s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177163518 (534h2m50.048246553s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177185680 (534h2m50.048268715s) [seg_pos={0 0}, upload=468370]: Segment reached optimal threshold of 13 pieces. Dec 27 21:31:50.177396331 (534h2m50.048479366s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection Dec 27 21:31:50.177403465 (534h2m50.0484865s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177520928 (534h2m50.048603963s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection Dec 27 21:31:50.177527100 (534h2m50.048610135s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177677287 (534h2m50.048760322s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 21:31:50.177683198 (534h2m50.048766243s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177702183 (534h2m50.048785218s) [seg_pos={0 0}, upload=468370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:31:50.177929246 (534h2m50.049012281s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:31:50.177935578 (534h2m50.049018623s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:31:50.472070343 (534h2m50.343153388s) [upload=468370]: Done uploading segments... Dec 27 21:31:50.472087114 (534h2m50.343170149s) [upload=468370]: Upload finished. err: <nil> Duration: 6.014913192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..XX..X....................X..........X......X........X...] duration: 6.080657859s trace: https://jaeger.dev.storj.io/trace/39d2c7777fcb15f8 FAIL (2025-12-27 21:34:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ) context deadline exceeded ======LOGS====== (534h2m47.392842023s) [upload=468370]: Uploading segments... Dec 27 21:31:47.521936445 (534h2m47.39301948s) [upload=468370]: Got next segment. Inline: false Dec 27 21:31:47.521974357 (534h2m47.393057393s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:31:47.521979097 (534h2m47.393062132s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:31:47.767045857 (534h2m47.638128902s) [seg_pos={0 0}, upload=468370]: Begin upload segment... Dec 27 21:31:47.767092054 (534h2m47.638175089s) [seg_pos={0 0}, upload=468370]: Done begin upload segment. Dec 27 21:31:47.767139495 (534h2m47.63822253s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767159122 (534h2m47.638242167s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767660247 (534h2m47.638743282s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767861990 (534h2m47.638945025s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768148817 (534h2m47.639231852s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768483103 (534h2m47.639566148s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768792382 (534h2m47.639875417s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769038832 (534h2m47.640121877s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769312453 (534h2m47.640395488s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769559002 (534h2m47.640642037s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769877889 (534h2m47.640960924s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770064996 (534h2m47.641148031s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770233427 (534h2m47.641316462s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771289227 (534h2m47.642372272s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771584158 (534h2m47.642667193s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771872227 (534h2m47.642955272s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:49.490998508 (534h2m49.362081543s) [upload=468370]: Next returned nil segment Dec 27 21:31:49.491016102 (534h2m49.362099137s) [upload=468370]: Waiting for error group managing segments... Dec 27 21:31:49.770955245 (534h2m49.64203829s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.771058903 (534h2m49.642141938s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.777122399 (534h2m49.648205434s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.790164273 (534h2m49.661247308s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.823247729 (534h2m49.694330764s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.826146289 (534h2m49.697229334s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.847994842 (534h2m49.719077887s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.996799172 (534h2m49.867882207s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047476210 (534h2m49.918559256s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047674247 (534h2m49.918757282s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.048428333 (534h2m49.919511378s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.067288637 (534h2m49.938371682s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177163518 (534h2m50.048246553s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177185680 (534h2m50.048268715s) [seg_pos={0 0}, upload=468370]: Segment reached optimal threshold of 13 pieces. Dec 27 21:31:50.177396331 (534h2m50.048479366s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection Dec 27 21:31:50.177403465 (534h2m50.0484865s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177520928 (534h2m50.048603963s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection Dec 27 21:31:50.177527100 (534h2m50.048610135s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177677287 (534h2m50.048760322s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 21:31:50.177683198 (534h2m50.048766243s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177702183 (534h2m50.048785218s) [seg_pos={0 0}, upload=468370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:31:50.177929246 (534h2m50.049012281s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:31:50.177935578 (534h2m50.049018623s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:31:50.472070343 (534h2m50.343153388s) [upload=468370]: Done uploading segments... Dec 27 21:31:50.472087114 (534h2m50.343170149s) [upload=468370]: Upload finished. err: <nil> Duration: 6.08069063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..XX..X....................X..........X......X........X..] duration: 5.962732481s trace: https://jaeger.dev.storj.io/trace/39d2c7777fcb15f8 FAIL (2025-12-27 21:33:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ) context deadline exceeded ======LOGS====== (534h2m47.392842023s) [upload=468370]: Uploading segments... Dec 27 21:31:47.521936445 (534h2m47.39301948s) [upload=468370]: Got next segment. Inline: false Dec 27 21:31:47.521974357 (534h2m47.393057393s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:31:47.521979097 (534h2m47.393062132s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:31:47.767045857 (534h2m47.638128902s) [seg_pos={0 0}, upload=468370]: Begin upload segment... Dec 27 21:31:47.767092054 (534h2m47.638175089s) [seg_pos={0 0}, upload=468370]: Done begin upload segment. Dec 27 21:31:47.767139495 (534h2m47.63822253s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767159122 (534h2m47.638242167s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767660247 (534h2m47.638743282s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767861990 (534h2m47.638945025s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768148817 (534h2m47.639231852s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768483103 (534h2m47.639566148s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768792382 (534h2m47.639875417s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769038832 (534h2m47.640121877s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769312453 (534h2m47.640395488s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769559002 (534h2m47.640642037s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769877889 (534h2m47.640960924s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770064996 (534h2m47.641148031s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770233427 (534h2m47.641316462s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771289227 (534h2m47.642372272s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771584158 (534h2m47.642667193s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771872227 (534h2m47.642955272s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:49.490998508 (534h2m49.362081543s) [upload=468370]: Next returned nil segment Dec 27 21:31:49.491016102 (534h2m49.362099137s) [upload=468370]: Waiting for error group managing segments... Dec 27 21:31:49.770955245 (534h2m49.64203829s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.771058903 (534h2m49.642141938s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.777122399 (534h2m49.648205434s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.790164273 (534h2m49.661247308s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.823247729 (534h2m49.694330764s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.826146289 (534h2m49.697229334s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.847994842 (534h2m49.719077887s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.996799172 (534h2m49.867882207s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047476210 (534h2m49.918559256s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047674247 (534h2m49.918757282s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.048428333 (534h2m49.919511378s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.067288637 (534h2m49.938371682s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177163518 (534h2m50.048246553s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177185680 (534h2m50.048268715s) [seg_pos={0 0}, upload=468370]: Segment reached optimal threshold of 13 pieces. Dec 27 21:31:50.177396331 (534h2m50.048479366s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection Dec 27 21:31:50.177403465 (534h2m50.0484865s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177520928 (534h2m50.048603963s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection Dec 27 21:31:50.177527100 (534h2m50.048610135s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177677287 (534h2m50.048760322s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 21:31:50.177683198 (534h2m50.048766243s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177702183 (534h2m50.048785218s) [seg_pos={0 0}, upload=468370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:31:50.177929246 (534h2m50.049012281s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:31:50.177935578 (534h2m50.049018623s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:31:50.472070343 (534h2m50.343153388s) [upload=468370]: Done uploading segments... Dec 27 21:31:50.472087114 (534h2m50.343170149s) [upload=468370]: Upload finished. err: <nil> Duration: 5.962747789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..XX..X....................X..........X......X........X.] duration: 6.015703285s trace: https://jaeger.dev.storj.io/trace/39d2c7777fcb15f8 FAIL (2025-12-27 21:32:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ) context deadline exceeded ======LOGS====== (534h2m47.392842023s) [upload=468370]: Uploading segments... Dec 27 21:31:47.521936445 (534h2m47.39301948s) [upload=468370]: Got next segment. Inline: false Dec 27 21:31:47.521974357 (534h2m47.393057393s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:31:47.521979097 (534h2m47.393062132s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:31:47.767045857 (534h2m47.638128902s) [seg_pos={0 0}, upload=468370]: Begin upload segment... Dec 27 21:31:47.767092054 (534h2m47.638175089s) [seg_pos={0 0}, upload=468370]: Done begin upload segment. Dec 27 21:31:47.767139495 (534h2m47.63822253s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767159122 (534h2m47.638242167s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767660247 (534h2m47.638743282s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767861990 (534h2m47.638945025s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768148817 (534h2m47.639231852s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768483103 (534h2m47.639566148s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768792382 (534h2m47.639875417s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769038832 (534h2m47.640121877s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769312453 (534h2m47.640395488s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769559002 (534h2m47.640642037s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769877889 (534h2m47.640960924s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770064996 (534h2m47.641148031s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770233427 (534h2m47.641316462s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771289227 (534h2m47.642372272s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771584158 (534h2m47.642667193s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771872227 (534h2m47.642955272s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:49.490998508 (534h2m49.362081543s) [upload=468370]: Next returned nil segment Dec 27 21:31:49.491016102 (534h2m49.362099137s) [upload=468370]: Waiting for error group managing segments... Dec 27 21:31:49.770955245 (534h2m49.64203829s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.771058903 (534h2m49.642141938s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.777122399 (534h2m49.648205434s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.790164273 (534h2m49.661247308s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.823247729 (534h2m49.694330764s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.826146289 (534h2m49.697229334s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.847994842 (534h2m49.719077887s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.996799172 (534h2m49.867882207s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047476210 (534h2m49.918559256s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047674247 (534h2m49.918757282s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.048428333 (534h2m49.919511378s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.067288637 (534h2m49.938371682s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177163518 (534h2m50.048246553s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177185680 (534h2m50.048268715s) [seg_pos={0 0}, upload=468370]: Segment reached optimal threshold of 13 pieces. Dec 27 21:31:50.177396331 (534h2m50.048479366s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection Dec 27 21:31:50.177403465 (534h2m50.0484865s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177520928 (534h2m50.048603963s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection Dec 27 21:31:50.177527100 (534h2m50.048610135s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177677287 (534h2m50.048760322s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 21:31:50.177683198 (534h2m50.048766243s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177702183 (534h2m50.048785218s) [seg_pos={0 0}, upload=468370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:31:50.177929246 (534h2m50.049012281s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:31:50.177935578 (534h2m50.049018623s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:31:50.472070343 (534h2m50.343153388s) [upload=468370]: Done uploading segments... Dec 27 21:31:50.472087114 (534h2m50.343170149s) [upload=468370]: Upload finished. err: <nil> Duration: 6.015735215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..XX..X....................X..........X......X........X] duration: 30.002229853s trace: https://jaeger.dev.storj.io/trace/39d2c7777fcb15f8 FAIL (2025-12-27 21:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ) context deadline exceeded ======LOGS====== (534h2m47.392842023s) [upload=468370]: Uploading segments... Dec 27 21:31:47.521936445 (534h2m47.39301948s) [upload=468370]: Got next segment. Inline: false Dec 27 21:31:47.521974357 (534h2m47.393057393s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:31:47.521979097 (534h2m47.393062132s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:31:47.767045857 (534h2m47.638128902s) [seg_pos={0 0}, upload=468370]: Begin upload segment... Dec 27 21:31:47.767092054 (534h2m47.638175089s) [seg_pos={0 0}, upload=468370]: Done begin upload segment. Dec 27 21:31:47.767139495 (534h2m47.63822253s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767159122 (534h2m47.638242167s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767660247 (534h2m47.638743282s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.767861990 (534h2m47.638945025s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768148817 (534h2m47.639231852s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768483103 (534h2m47.639566148s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.768792382 (534h2m47.639875417s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769038832 (534h2m47.640121877s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769312453 (534h2m47.640395488s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769559002 (534h2m47.640642037s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.769877889 (534h2m47.640960924s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770064996 (534h2m47.641148031s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.770233427 (534h2m47.641316462s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771289227 (534h2m47.642372272s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771584158 (534h2m47.642667193s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:47.771872227 (534h2m47.642955272s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Uploading piece... Dec 27 21:31:49.490998508 (534h2m49.362081543s) [upload=468370]: Next returned nil segment Dec 27 21:31:49.491016102 (534h2m49.362099137s) [upload=468370]: Waiting for error group managing segments... Dec 27 21:31:49.770955245 (534h2m49.64203829s) [noise=true , address=145.79.227.42:4122 , piece_id=ISWVSWZC5MFORIWH3RJEBLE7HCJQKR4HR2Q4G5YNZOQHT3BAI6SQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.771058903 (534h2m49.642141938s) [noise=true , address=145.79.227.15:4876 , piece_id=NHWSMBLGDAO7VJN4QGM3VQPYRNKEYNQRK77NXBQAR7YYKQ7JERBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.777122399 (534h2m49.648205434s) [noise=true , address=145.79.227.10:4402 , piece_id=M6TLAQLAILQA2JCNHTGCS6V4FHM2R3H4RF4YX7VBGNNGMQGIMCHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.790164273 (534h2m49.661247308s) [noise=true , address=145.79.227.48:4303 , piece_id=OGTXXHAREEZN6J6ND2SDUQHIYHIW4GFZK6BIKMSU2V34JFAMSZNA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.823247729 (534h2m49.694330764s) [noise=true , address=145.79.227.25:4618 , piece_id=ASHZO4KLO4T3XNDUAW2I4GOLLH2CV5AOHSONLKZEJ2UGC24PB7AQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.826146289 (534h2m49.697229334s) [noise=true , address=145.79.227.21:4474 , piece_id=Y2UJVEY6BKXL4FSY5G65YU5C2UC4AOX7KKOZQDDIUSV34VZ4Q3WA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.847994842 (534h2m49.719077887s) [noise=true , address=145.79.227.38:4977 , piece_id=WCJ6ACGI5GYLT2G6GWMINIMH4ZCMSPKIJDL6JTPGCYYHXIXDO62Q, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:49.996799172 (534h2m49.867882207s) [noise=true , address=45.91.116.6:4974 , piece_id=3FCIWQPYJDATJAHKITY2HGAWU2Y5X5LSMGEO7F6TYBQTRBFRQBTA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047476210 (534h2m49.918559256s) [noise=true , address=145.79.227.39:4013 , piece_id=QZEPCACNXTQHRLSFPL5EABFYUHJKD4E7OG7KCWPUULL2FNS67ZHA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.047674247 (534h2m49.918757282s) [noise=true , address=45.91.116.14:4191 , piece_id=WJRDMQMVPKTFYLRFSDPFJKCZUT62TE3HYY45ZONUQMT3BWL4XMKQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.048428333 (534h2m49.919511378s) [noise=true , address=45.91.116.5:4299 , piece_id=NIENM4BS5MVLYCIETG4526B4A2JU6V5DVM45BJG357UPIXH7MXAQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.067288637 (534h2m49.938371682s) [noise=true , address=77.90.31.19:4842 , piece_id=WLCHJBCGTCNI2THQNXDUGG545BIOYKSXKV6FPVH36BKEQ3V25SNQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177163518 (534h2m50.048246553s) [noise=true , address=45.91.116.10:4084 , piece_id=E5I2GFLJ5M3DRFIY5B7K2Q2GN6I7NNNM6NGL2KTXK43S62VTC7LA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: <nil> Dec 27 21:31:50.177185680 (534h2m50.048268715s) [seg_pos={0 0}, upload=468370]: Segment reached optimal threshold of 13 pieces. Dec 27 21:31:50.177396331 (534h2m50.048479366s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41982->77.90.31.21:4880: use of closed network connection Dec 27 21:31:50.177403465 (534h2m50.0484865s) [noise=true , address=77.90.31.21:4880 , piece_id=KQI7RA2MTXDDRK5TE6CPWBRIXOPWA4EA745N7GR3LPK45PBSJXBA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177520928 (534h2m50.048603963s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46908->77.90.31.15:4700: use of closed network connection Dec 27 21:31:50.177527100 (534h2m50.048610135s) [noise=true , address=77.90.31.15:4700 , piece_id=FHMQQHKTAGAKVE5TH5O5LYY2QPQXNF6HLVXQ56SDXPBRG2YJ3KPA, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177677287 (534h2m50.048760322s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 21:31:50.177683198 (534h2m50.048766243s) [noise=true , address=145.79.227.33:4871 , piece_id=WJM7WRFP37LPCBZBNDNWVBTM3VHJCUXVE7J4PVSV5IFXWMQH36ZQ, seg_pos={0 0}, upload=468370]: Long tail cancellation due to optimal threshold Dec 27 21:31:50.177702183 (534h2m50.048785218s) [seg_pos={0 0}, upload=468370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:31:50.177929246 (534h2m50.049012281s) [upload=468370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:31:50.177935578 (534h2m50.049018623s) [upload=468370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:31:50.472070343 (534h2m50.343153388s) [upload=468370]: Done uploading segments... Dec 27 21:31:50.472087114 (534h2m50.343170149s) [upload=468370]: Upload finished. err: <nil> Duration: 30.002249631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X....X..XX..X....................X..........X......X........] duration: 5.837540161s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 5.837563927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X....X..XX..X....................X..........X......X.......] duration: 6.556603667s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:29:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 6.556628393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X....X..XX..X....................X..........X......X......] duration: 5.972134793s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 5.972160902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X....X..XX..X....................X..........X......X.....] duration: 13.537408426s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:28:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 13.537435839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X....X..XX..X....................X..........X......X....] duration: 6.374221348s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:27:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 6.374236206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X....X..XX..X....................X..........X......X...] duration: 6.064328396s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:25:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 6.064359486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X....X..XX..X....................X..........X......X..] duration: 5.753580006s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:25:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 5.753596357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X....X..XX..X....................X..........X......X.] duration: 5.926045771s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:25:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 5.926074377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X....X..XX..X....................X..........X......X] duration: 30.002215057s trace: https://jaeger.dev.storj.io/trace/42a50cd935121196 FAIL (2025-12-27 21:24:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ) context deadline exceeded ======LOGS====== Dec 27 21:24:24.284934948 (533h55m24.156017983s) [upload=468263]: Starting upload Dec 27 21:24:24.284988630 (533h55m24.156071665s) [upload=468263]: Done starting upload Dec 27 21:24:24.285066790 (533h55m24.156149825s) [upload=468263]: Uploading segments... Dec 27 21:24:24.285202407 (533h55m24.156285442s) [upload=468263]: Got next segment. Inline: false Dec 27 21:24:24.285240661 (533h55m24.156323696s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:24:24.285244648 (533h55m24.156327683s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:24:24.530445964 (533h55m24.401528999s) [seg_pos={0 0}, upload=468263]: Begin upload segment... Dec 27 21:24:24.530488947 (533h55m24.401571982s) [seg_pos={0 0}, upload=468263]: Done begin upload segment. Dec 27 21:24:24.530514545 (533h55m24.40159759s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.530606179 (533h55m24.401689214s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531163330 (533h55m24.402246365s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531239004 (533h55m24.402322039s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531488900 (533h55m24.402571935s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531629978 (533h55m24.402713014s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531776178 (533h55m24.402859223s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.531986558 (533h55m24.403069593s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532064486 (533h55m24.403147521s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532312889 (533h55m24.403395924s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532341153 (533h55m24.403424188s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532558587 (533h55m24.403641622s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532769749 (533h55m24.403852784s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.532805677 (533h55m24.403888713s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533253320 (533h55m24.404336355s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:24.533602365 (533h55m24.4046854s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Uploading piece... Dec 27 21:24:26.242974381 (533h55m26.114057416s) [upload=468263]: Next returned nil segment Dec 27 21:24:26.242987026 (533h55m26.114070061s) [upload=468263]: Waiting for error group managing segments... Dec 27 21:24:26.540993804 (533h55m26.412076839s) [noise=true , address=145.79.227.36:4517 , piece_id=FZC7M7B5PHKZUIWP233VJ4OXYI6AOADBCC4TAC6PBOJCJFUP76EQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548092290 (533h55m26.419175335s) [noise=true , address=145.79.227.39:4015 , piece_id=S63NDWE2YVBIVEUEVZ6RXL56I74WEYQBIAKR2LEHWWXQYSXYBSEA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.548569178 (533h55m26.419652213s) [noise=true , address=145.79.227.15:4876 , piece_id=CFIXEFUE5YOLSDR4GJGIV4SDCUBZ2LKGVFK4DRLRBPPFM2D5FLBA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549167899 (533h55m26.420250934s) [noise=true , address=145.79.227.52:4447 , piece_id=5WHCOCYFT3RPK73HPKDHGEJDUJOL2WBB6KE7V6WBIZ3WP7K35KQQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.549566748 (533h55m26.420649783s) [noise=true , address=145.79.227.45:4229 , piece_id=MEC3LWRN46PTNV27F3ZU6OUTKBLUYXJ4SIY4VZ3QP5DPGEPDD6CA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.550617779 (533h55m26.421700824s) [noise=true , address=145.79.227.38:4978 , piece_id=2UDPJAH3BXVMUWIVNAGTYRRJ6G7SBHYGJCY42NS5E4XYFLMBMQRQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.562973236 (533h55m26.434056271s) [noise=true , address=145.79.227.28:4727 , piece_id=BAL2FEJCXVJQMON6S5A2P2FNN4V2PK3XLQU4UEONV7J4W3ONAUUA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.563423383 (533h55m26.434506418s) [noise=true , address=145.79.227.8:4329 , piece_id=XNEAELWMFLLTYG4DOYXBGQIVCYIG23JSGCZIMKWQO3VKRFZ2Q2HA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.568376164 (533h55m26.43945921s) [noise=true , address=145.79.227.5:4223 , piece_id=YXN2RMFXIJVUEMYWMCYKU2MIHWEMRXOPBZ7LEYXAAS5BU23JKMPQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.774868853 (533h55m26.645951888s) [noise=true , address=45.91.116.6:4975 , piece_id=XY4HFCJ2TZBIIENBJPPSOGQ6F436FTZVOOZPMXVOAK53366YNMRA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.783685340 (533h55m26.654768385s) [noise=true , address=45.91.116.9:4261 , piece_id=M5ISDYZHKQUVO5X2WZAIA72JQSGP5P7FCIG7LWUGIPKMI67ZMUCA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.841334726 (533h55m26.712417761s) [noise=true , address=45.91.116.5:4297 , piece_id=KXMDSSQFEUV4YBQZPVCOPTG3MCSPRE7UQMSVLN7NL2ZKX4XOCTOQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868859438 (533h55m26.739942473s) [noise=true , address=45.91.116.8:4046 , piece_id=3AIC2Z55EMETT67Y7ERFJU6INYROTPCA7GDA77PBYOUB3NFIC5TQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: <nil> Dec 27 21:24:26.868878554 (533h55m26.739961599s) [seg_pos={0 0}, upload=468263]: Segment reached optimal threshold of 13 pieces. Dec 27 21:24:26.869072193 (533h55m26.740155228s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869078324 (533h55m26.740161359s) [noise=true , address=77.90.31.11:4553 , piece_id=CR3S5K2Y26O5MBGPMNVFGPKJ4HB4VA6LEP52F54TDTLB4FKDRDUQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869179136 (533h55m26.740262171s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869184567 (533h55m26.740267602s) [noise=true , address=145.79.227.50:4373 , piece_id=DJH2OIV7Z655FPIK5YIVDZD5CCMWFI2CPV3F74UEXFDKER75EAWQ, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869301028 (533h55m26.740384073s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 27 21:24:26.869306589 (533h55m26.740389624s) [noise=true , address=145.79.227.29:4943 , piece_id=BPDABB23RZCHBMT2GIUNBKUMUKHWRL5MIS25NTOHUPJ6S4NDW2WA, seg_pos={0 0}, upload=468263]: Long tail cancellation due to optimal threshold Dec 27 21:24:26.869330544 (533h55m26.740413579s) [seg_pos={0 0}, upload=468263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:24:26.869397812 (533h55m26.740480848s) [upload=468263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:24:26.869404184 (533h55m26.74048722s) [upload=468263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:24:27.161039781 (533h55m27.032122826s) [upload=468263]: Done uploading segments... Dec 27 21:24:27.161061022 (533h55m27.032144057s) [upload=468263]: Upload finished. err: <nil> Duration: 30.002236427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X....X..XX..X....................X..........X......] duration: 5.919604867s trace: https://jaeger.dev.storj.io/trace/299212f7bfd9d218 FAIL (2025-12-27 21:23:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA) context deadline exceeded ======LOGS====== Dec 27 21:18:06.453005308 (533h49m6.324088353s) [upload=468173]: Starting upload Dec 27 21:18:06.453098215 (533h49m6.324181271s) [upload=468173]: Done starting upload Dec 27 21:18:06.453186723 (533h49m6.324269769s) [upload=468173]: Uploading segments... Dec 27 21:18:06.453417373 (533h49m6.324500408s) [upload=468173]: Got next segment. Inline: false Dec 27 21:18:06.453474702 (533h49m6.324557737s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:18:06.453480964 (533h49m6.324564009s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:18:06.698104691 (533h49m6.569187736s) [seg_pos={0 0}, upload=468173]: Begin upload segment... Dec 27 21:18:06.698163161 (533h49m6.569246197s) [seg_pos={0 0}, upload=468173]: Done begin upload segment. Dec 27 21:18:06.698231912 (533h49m6.569314957s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.698246080 (533h49m6.569329115s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699012479 (533h49m6.570095514s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699032647 (533h49m6.570115682s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699430736 (533h49m6.570513771s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699497192 (533h49m6.570580237s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699815729 (533h49m6.570898774s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700221702 (533h49m6.571304747s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700588700 (533h49m6.571671746s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700968564 (533h49m6.572051599s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701443869 (533h49m6.572526914s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701872746 (533h49m6.572955781s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702399539 (533h49m6.573482574s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702790603 (533h49m6.573873648s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703246843 (533h49m6.574329878s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703815044 (533h49m6.574898079s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:08.420091297 (533h49m8.291174343s) [upload=468173]: Next returned nil segment Dec 27 21:18:08.420117898 (533h49m8.291200943s) [upload=468173]: Waiting for error group managing segments... Dec 27 21:18:08.702818209 (533h49m8.573901245s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.710164268 (533h49m8.581247303s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.711059122 (533h49m8.582142167s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.716900014 (533h49m8.587983079s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.727924276 (533h49m8.599007311s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733370607 (533h49m8.604453662s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733619691 (533h49m8.604702736s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.736483774 (533h49m8.60756682s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.909301199 (533h49m8.780384234s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.923614786 (533h49m8.794697821s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.930728190 (533h49m8.801811255s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.008821805 (533h49m8.87990484s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077229294 (533h49m8.948312339s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077260103 (533h49m8.948343148s) [seg_pos={0 0}, upload=468173]: Segment reached optimal threshold of 13 pieces. Dec 27 21:18:09.077474912 (533h49m8.948557947s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077481675 (533h49m8.94856472s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077637362 (533h49m8.948720397s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077646509 (533h49m8.948729544s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077675063 (533h49m8.948758098s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077681806 (533h49m8.948764841s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077698006 (533h49m8.948781041s) [seg_pos={0 0}, upload=468173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:18:09.077764493 (533h49m8.948847538s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:18:09.077908978 (533h49m8.948992053s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:18:09.370129449 (533h49m9.241212494s) [upload=468173]: Done uploading segments... Dec 27 21:18:09.370156701 (533h49m9.241239746s) [upload=468173]: Upload finished. err: <nil> Duration: 5.91978977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....X..XX..X....................X..........X.....] duration: 6.14537266s trace: https://jaeger.dev.storj.io/trace/299212f7bfd9d218 FAIL (2025-12-27 21:23:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA) context deadline exceeded ======LOGS====== Dec 27 21:18:06.453005308 (533h49m6.324088353s) [upload=468173]: Starting upload Dec 27 21:18:06.453098215 (533h49m6.324181271s) [upload=468173]: Done starting upload Dec 27 21:18:06.453186723 (533h49m6.324269769s) [upload=468173]: Uploading segments... Dec 27 21:18:06.453417373 (533h49m6.324500408s) [upload=468173]: Got next segment. Inline: false Dec 27 21:18:06.453474702 (533h49m6.324557737s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:18:06.453480964 (533h49m6.324564009s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:18:06.698104691 (533h49m6.569187736s) [seg_pos={0 0}, upload=468173]: Begin upload segment... Dec 27 21:18:06.698163161 (533h49m6.569246197s) [seg_pos={0 0}, upload=468173]: Done begin upload segment. Dec 27 21:18:06.698231912 (533h49m6.569314957s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.698246080 (533h49m6.569329115s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699012479 (533h49m6.570095514s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699032647 (533h49m6.570115682s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699430736 (533h49m6.570513771s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699497192 (533h49m6.570580237s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699815729 (533h49m6.570898774s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700221702 (533h49m6.571304747s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700588700 (533h49m6.571671746s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700968564 (533h49m6.572051599s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701443869 (533h49m6.572526914s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701872746 (533h49m6.572955781s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702399539 (533h49m6.573482574s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702790603 (533h49m6.573873648s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703246843 (533h49m6.574329878s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703815044 (533h49m6.574898079s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:08.420091297 (533h49m8.291174343s) [upload=468173]: Next returned nil segment Dec 27 21:18:08.420117898 (533h49m8.291200943s) [upload=468173]: Waiting for error group managing segments... Dec 27 21:18:08.702818209 (533h49m8.573901245s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.710164268 (533h49m8.581247303s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.711059122 (533h49m8.582142167s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.716900014 (533h49m8.587983079s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.727924276 (533h49m8.599007311s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733370607 (533h49m8.604453662s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733619691 (533h49m8.604702736s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.736483774 (533h49m8.60756682s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.909301199 (533h49m8.780384234s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.923614786 (533h49m8.794697821s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.930728190 (533h49m8.801811255s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.008821805 (533h49m8.87990484s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077229294 (533h49m8.948312339s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077260103 (533h49m8.948343148s) [seg_pos={0 0}, upload=468173]: Segment reached optimal threshold of 13 pieces. Dec 27 21:18:09.077474912 (533h49m8.948557947s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077481675 (533h49m8.94856472s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077637362 (533h49m8.948720397s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077646509 (533h49m8.948729544s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077675063 (533h49m8.948758098s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077681806 (533h49m8.948764841s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077698006 (533h49m8.948781041s) [seg_pos={0 0}, upload=468173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:18:09.077764493 (533h49m8.948847538s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:18:09.077908978 (533h49m8.948992053s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:18:09.370129449 (533h49m9.241212494s) [upload=468173]: Done uploading segments... Dec 27 21:18:09.370156701 (533h49m9.241239746s) [upload=468173]: Upload finished. err: <nil> Duration: 6.145402936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X....X..XX..X....................X..........X....] duration: 7.173976051s trace: https://jaeger.dev.storj.io/trace/299212f7bfd9d218 FAIL (2025-12-27 21:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA) context deadline exceeded ======LOGS====== Dec 27 21:18:06.453005308 (533h49m6.324088353s) [upload=468173]: Starting upload Dec 27 21:18:06.453098215 (533h49m6.324181271s) [upload=468173]: Done starting upload Dec 27 21:18:06.453186723 (533h49m6.324269769s) [upload=468173]: Uploading segments... Dec 27 21:18:06.453417373 (533h49m6.324500408s) [upload=468173]: Got next segment. Inline: false Dec 27 21:18:06.453474702 (533h49m6.324557737s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:18:06.453480964 (533h49m6.324564009s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:18:06.698104691 (533h49m6.569187736s) [seg_pos={0 0}, upload=468173]: Begin upload segment... Dec 27 21:18:06.698163161 (533h49m6.569246197s) [seg_pos={0 0}, upload=468173]: Done begin upload segment. Dec 27 21:18:06.698231912 (533h49m6.569314957s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.698246080 (533h49m6.569329115s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699012479 (533h49m6.570095514s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699032647 (533h49m6.570115682s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699430736 (533h49m6.570513771s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699497192 (533h49m6.570580237s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699815729 (533h49m6.570898774s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700221702 (533h49m6.571304747s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700588700 (533h49m6.571671746s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700968564 (533h49m6.572051599s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701443869 (533h49m6.572526914s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701872746 (533h49m6.572955781s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702399539 (533h49m6.573482574s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702790603 (533h49m6.573873648s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703246843 (533h49m6.574329878s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703815044 (533h49m6.574898079s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:08.420091297 (533h49m8.291174343s) [upload=468173]: Next returned nil segment Dec 27 21:18:08.420117898 (533h49m8.291200943s) [upload=468173]: Waiting for error group managing segments... Dec 27 21:18:08.702818209 (533h49m8.573901245s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.710164268 (533h49m8.581247303s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.711059122 (533h49m8.582142167s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.716900014 (533h49m8.587983079s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.727924276 (533h49m8.599007311s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733370607 (533h49m8.604453662s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733619691 (533h49m8.604702736s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.736483774 (533h49m8.60756682s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.909301199 (533h49m8.780384234s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.923614786 (533h49m8.794697821s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.930728190 (533h49m8.801811255s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.008821805 (533h49m8.87990484s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077229294 (533h49m8.948312339s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077260103 (533h49m8.948343148s) [seg_pos={0 0}, upload=468173]: Segment reached optimal threshold of 13 pieces. Dec 27 21:18:09.077474912 (533h49m8.948557947s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077481675 (533h49m8.94856472s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077637362 (533h49m8.948720397s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077646509 (533h49m8.948729544s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077675063 (533h49m8.948758098s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077681806 (533h49m8.948764841s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077698006 (533h49m8.948781041s) [seg_pos={0 0}, upload=468173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:18:09.077764493 (533h49m8.948847538s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:18:09.077908978 (533h49m8.948992053s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:18:09.370129449 (533h49m9.241212494s) [upload=468173]: Done uploading segments... Dec 27 21:18:09.370156701 (533h49m9.241239746s) [upload=468173]: Upload finished. err: <nil> Duration: 7.174002462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X....X..XX..X....................X..........X...] duration: 6.174084667s trace: https://jaeger.dev.storj.io/trace/299212f7bfd9d218 FAIL (2025-12-27 21:20:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA) context deadline exceeded ======LOGS====== Dec 27 21:18:06.453005308 (533h49m6.324088353s) [upload=468173]: Starting upload Dec 27 21:18:06.453098215 (533h49m6.324181271s) [upload=468173]: Done starting upload Dec 27 21:18:06.453186723 (533h49m6.324269769s) [upload=468173]: Uploading segments... Dec 27 21:18:06.453417373 (533h49m6.324500408s) [upload=468173]: Got next segment. Inline: false Dec 27 21:18:06.453474702 (533h49m6.324557737s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:18:06.453480964 (533h49m6.324564009s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:18:06.698104691 (533h49m6.569187736s) [seg_pos={0 0}, upload=468173]: Begin upload segment... Dec 27 21:18:06.698163161 (533h49m6.569246197s) [seg_pos={0 0}, upload=468173]: Done begin upload segment. Dec 27 21:18:06.698231912 (533h49m6.569314957s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.698246080 (533h49m6.569329115s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699012479 (533h49m6.570095514s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699032647 (533h49m6.570115682s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699430736 (533h49m6.570513771s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699497192 (533h49m6.570580237s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699815729 (533h49m6.570898774s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700221702 (533h49m6.571304747s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700588700 (533h49m6.571671746s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700968564 (533h49m6.572051599s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701443869 (533h49m6.572526914s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701872746 (533h49m6.572955781s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702399539 (533h49m6.573482574s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702790603 (533h49m6.573873648s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703246843 (533h49m6.574329878s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703815044 (533h49m6.574898079s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:08.420091297 (533h49m8.291174343s) [upload=468173]: Next returned nil segment Dec 27 21:18:08.420117898 (533h49m8.291200943s) [upload=468173]: Waiting for error group managing segments... Dec 27 21:18:08.702818209 (533h49m8.573901245s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.710164268 (533h49m8.581247303s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.711059122 (533h49m8.582142167s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.716900014 (533h49m8.587983079s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.727924276 (533h49m8.599007311s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733370607 (533h49m8.604453662s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733619691 (533h49m8.604702736s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.736483774 (533h49m8.60756682s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.909301199 (533h49m8.780384234s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.923614786 (533h49m8.794697821s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.930728190 (533h49m8.801811255s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.008821805 (533h49m8.87990484s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077229294 (533h49m8.948312339s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077260103 (533h49m8.948343148s) [seg_pos={0 0}, upload=468173]: Segment reached optimal threshold of 13 pieces. Dec 27 21:18:09.077474912 (533h49m8.948557947s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077481675 (533h49m8.94856472s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077637362 (533h49m8.948720397s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077646509 (533h49m8.948729544s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077675063 (533h49m8.948758098s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077681806 (533h49m8.948764841s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077698006 (533h49m8.948781041s) [seg_pos={0 0}, upload=468173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:18:09.077764493 (533h49m8.948847538s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:18:09.077908978 (533h49m8.948992053s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:18:09.370129449 (533h49m9.241212494s) [upload=468173]: Done uploading segments... Dec 27 21:18:09.370156701 (533h49m9.241239746s) [upload=468173]: Upload finished. err: <nil> Duration: 6.17411782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X............X....X..XX..X....................X..........X..] duration: 5.697881173s trace: https://jaeger.dev.storj.io/trace/299212f7bfd9d218 FAIL (2025-12-27 21:19:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA) context deadline exceeded ======LOGS====== Dec 27 21:18:06.453005308 (533h49m6.324088353s) [upload=468173]: Starting upload Dec 27 21:18:06.453098215 (533h49m6.324181271s) [upload=468173]: Done starting upload Dec 27 21:18:06.453186723 (533h49m6.324269769s) [upload=468173]: Uploading segments... Dec 27 21:18:06.453417373 (533h49m6.324500408s) [upload=468173]: Got next segment. Inline: false Dec 27 21:18:06.453474702 (533h49m6.324557737s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:18:06.453480964 (533h49m6.324564009s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:18:06.698104691 (533h49m6.569187736s) [seg_pos={0 0}, upload=468173]: Begin upload segment... Dec 27 21:18:06.698163161 (533h49m6.569246197s) [seg_pos={0 0}, upload=468173]: Done begin upload segment. Dec 27 21:18:06.698231912 (533h49m6.569314957s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.698246080 (533h49m6.569329115s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699012479 (533h49m6.570095514s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699032647 (533h49m6.570115682s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699430736 (533h49m6.570513771s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699497192 (533h49m6.570580237s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699815729 (533h49m6.570898774s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700221702 (533h49m6.571304747s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700588700 (533h49m6.571671746s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700968564 (533h49m6.572051599s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701443869 (533h49m6.572526914s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701872746 (533h49m6.572955781s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702399539 (533h49m6.573482574s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702790603 (533h49m6.573873648s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703246843 (533h49m6.574329878s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703815044 (533h49m6.574898079s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:08.420091297 (533h49m8.291174343s) [upload=468173]: Next returned nil segment Dec 27 21:18:08.420117898 (533h49m8.291200943s) [upload=468173]: Waiting for error group managing segments... Dec 27 21:18:08.702818209 (533h49m8.573901245s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.710164268 (533h49m8.581247303s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.711059122 (533h49m8.582142167s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.716900014 (533h49m8.587983079s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.727924276 (533h49m8.599007311s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733370607 (533h49m8.604453662s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733619691 (533h49m8.604702736s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.736483774 (533h49m8.60756682s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.909301199 (533h49m8.780384234s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.923614786 (533h49m8.794697821s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.930728190 (533h49m8.801811255s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.008821805 (533h49m8.87990484s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077229294 (533h49m8.948312339s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077260103 (533h49m8.948343148s) [seg_pos={0 0}, upload=468173]: Segment reached optimal threshold of 13 pieces. Dec 27 21:18:09.077474912 (533h49m8.948557947s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077481675 (533h49m8.94856472s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077637362 (533h49m8.948720397s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077646509 (533h49m8.948729544s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077675063 (533h49m8.948758098s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077681806 (533h49m8.948764841s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077698006 (533h49m8.948781041s) [seg_pos={0 0}, upload=468173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:18:09.077764493 (533h49m8.948847538s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:18:09.077908978 (533h49m8.948992053s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:18:09.370129449 (533h49m9.241212494s) [upload=468173]: Done uploading segments... Dec 27 21:18:09.370156701 (533h49m9.241239746s) [upload=468173]: Upload finished. err: <nil> Duration: 5.697898405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X............X....X..XX..X....................X..........X.] duration: 6.059451917s trace: https://jaeger.dev.storj.io/trace/299212f7bfd9d218 FAIL (2025-12-27 21:18:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA) context deadline exceeded ======LOGS====== Dec 27 21:18:06.453005308 (533h49m6.324088353s) [upload=468173]: Starting upload Dec 27 21:18:06.453098215 (533h49m6.324181271s) [upload=468173]: Done starting upload Dec 27 21:18:06.453186723 (533h49m6.324269769s) [upload=468173]: Uploading segments... Dec 27 21:18:06.453417373 (533h49m6.324500408s) [upload=468173]: Got next segment. Inline: false Dec 27 21:18:06.453474702 (533h49m6.324557737s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:18:06.453480964 (533h49m6.324564009s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:18:06.698104691 (533h49m6.569187736s) [seg_pos={0 0}, upload=468173]: Begin upload segment... Dec 27 21:18:06.698163161 (533h49m6.569246197s) [seg_pos={0 0}, upload=468173]: Done begin upload segment. Dec 27 21:18:06.698231912 (533h49m6.569314957s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.698246080 (533h49m6.569329115s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699012479 (533h49m6.570095514s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699032647 (533h49m6.570115682s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699430736 (533h49m6.570513771s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699497192 (533h49m6.570580237s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699815729 (533h49m6.570898774s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700221702 (533h49m6.571304747s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700588700 (533h49m6.571671746s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700968564 (533h49m6.572051599s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701443869 (533h49m6.572526914s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701872746 (533h49m6.572955781s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702399539 (533h49m6.573482574s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702790603 (533h49m6.573873648s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703246843 (533h49m6.574329878s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703815044 (533h49m6.574898079s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:08.420091297 (533h49m8.291174343s) [upload=468173]: Next returned nil segment Dec 27 21:18:08.420117898 (533h49m8.291200943s) [upload=468173]: Waiting for error group managing segments... Dec 27 21:18:08.702818209 (533h49m8.573901245s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.710164268 (533h49m8.581247303s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.711059122 (533h49m8.582142167s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.716900014 (533h49m8.587983079s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.727924276 (533h49m8.599007311s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733370607 (533h49m8.604453662s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733619691 (533h49m8.604702736s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.736483774 (533h49m8.60756682s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.909301199 (533h49m8.780384234s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.923614786 (533h49m8.794697821s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.930728190 (533h49m8.801811255s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.008821805 (533h49m8.87990484s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077229294 (533h49m8.948312339s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077260103 (533h49m8.948343148s) [seg_pos={0 0}, upload=468173]: Segment reached optimal threshold of 13 pieces. Dec 27 21:18:09.077474912 (533h49m8.948557947s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077481675 (533h49m8.94856472s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077637362 (533h49m8.948720397s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077646509 (533h49m8.948729544s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077675063 (533h49m8.948758098s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077681806 (533h49m8.948764841s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077698006 (533h49m8.948781041s) [seg_pos={0 0}, upload=468173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:18:09.077764493 (533h49m8.948847538s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:18:09.077908978 (533h49m8.948992053s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:18:09.370129449 (533h49m9.241212494s) [upload=468173]: Done uploading segments... Dec 27 21:18:09.370156701 (533h49m9.241239746s) [upload=468173]: Upload finished. err: <nil> Duration: 6.059477365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X............X....X..XX..X....................X..........X] duration: 30.003354337s trace: https://jaeger.dev.storj.io/trace/299212f7bfd9d218 FAIL (2025-12-27 21:18:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA) context deadline exceeded ======LOGS====== Dec 27 21:18:06.453005308 (533h49m6.324088353s) [upload=468173]: Starting upload Dec 27 21:18:06.453098215 (533h49m6.324181271s) [upload=468173]: Done starting upload Dec 27 21:18:06.453186723 (533h49m6.324269769s) [upload=468173]: Uploading segments... Dec 27 21:18:06.453417373 (533h49m6.324500408s) [upload=468173]: Got next segment. Inline: false Dec 27 21:18:06.453474702 (533h49m6.324557737s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:18:06.453480964 (533h49m6.324564009s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:18:06.698104691 (533h49m6.569187736s) [seg_pos={0 0}, upload=468173]: Begin upload segment... Dec 27 21:18:06.698163161 (533h49m6.569246197s) [seg_pos={0 0}, upload=468173]: Done begin upload segment. Dec 27 21:18:06.698231912 (533h49m6.569314957s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.698246080 (533h49m6.569329115s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699012479 (533h49m6.570095514s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699032647 (533h49m6.570115682s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699430736 (533h49m6.570513771s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699497192 (533h49m6.570580237s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.699815729 (533h49m6.570898774s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700221702 (533h49m6.571304747s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700588700 (533h49m6.571671746s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.700968564 (533h49m6.572051599s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701443869 (533h49m6.572526914s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.701872746 (533h49m6.572955781s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702399539 (533h49m6.573482574s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.702790603 (533h49m6.573873648s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703246843 (533h49m6.574329878s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:06.703815044 (533h49m6.574898079s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Uploading piece... Dec 27 21:18:08.420091297 (533h49m8.291174343s) [upload=468173]: Next returned nil segment Dec 27 21:18:08.420117898 (533h49m8.291200943s) [upload=468173]: Waiting for error group managing segments... Dec 27 21:18:08.702818209 (533h49m8.573901245s) [noise=true , address=145.79.227.9:4367 , piece_id=YK6X3OOMZOA6VNWSKWOQQ5BZSBHPBAWTHXGEFNVNNVCFIOUJGE3Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.710164268 (533h49m8.581247303s) [noise=true , address=145.79.227.8:4329 , piece_id=5VGZWBQOGRHZ5OJMDDB76IXYXOGWVPBGQKLI6OSUOJZEBXSDLZGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.711059122 (533h49m8.582142167s) [noise=true , address=145.79.227.52:4445 , piece_id=CBULHQL3VEEYXT4E6FLX6VXEYHSWO7IW4AI34N2BEFVL6HW52GPA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.716900014 (533h49m8.587983079s) [noise=true , address=145.79.227.23:4547 , piece_id=NRV2PZKMZUMRG3JUPZ2GBBAUI4B4T4OVVKTASIMWFLQS5K2G2Y3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.727924276 (533h49m8.599007311s) [noise=true , address=145.79.227.21:4475 , piece_id=UO5EDVVM4VYTYDR2AKZAHSSVGQJ5IHMWETYGNBETYSUH65PQ56WA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733370607 (533h49m8.604453662s) [noise=true , address=145.79.227.15:4873 , piece_id=WC6535LVOHVXULPBWRKDIOMUBT3PXKBDAZV7YHJSJLYYGABBVBPQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.733619691 (533h49m8.604702736s) [noise=true , address=145.79.227.50:4375 , piece_id=6SVONEA4DU26QA6A3MWPK7FLZ7FUFMJC5KHWRGIU7ICL63HZNT7Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.736483774 (533h49m8.60756682s) [noise=true , address=145.79.227.47:4266 , piece_id=4EAGKKXV3OFCSCO6ZYV2JTIGTGH37SQTDV3P6SRSGUVWO3OU7JGA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.909301199 (533h49m8.780384234s) [noise=true , address=77.90.31.10:4519 , piece_id=P7SHZ7BMN7HZGVAIMVZQALEMGWCVHOB5SNT45X4XYVJWKENIOBVQ, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.923614786 (533h49m8.794697821s) [noise=true , address=45.91.116.13:4153 , piece_id=CYEZBUOV7GY3NYP5TSWXWN3DGU4UOSPEBWS3UREQLWWNE26DQI6Q, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:08.930728190 (533h49m8.801811255s) [noise=true , address=77.90.31.11:4554 , piece_id=LLHT6YURKVAD5GDYGR4FE3E7TEMAKXNNWBXR33TZLZNJ56Y5SOYA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.008821805 (533h49m8.87990484s) [noise=true , address=45.91.116.6:4974 , piece_id=NUBVXIQO73BR3VVBCQ7M5HXYMHPC5ITCXG4FZAIZLWNPJJ6PPBEA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077229294 (533h49m8.948312339s) [noise=true , address=45.91.116.4:4937 , piece_id=P74HZVV5PUZC2GMHDSGN535BFI5NJZ4OIWVDLVHN2GYBST2GBYCA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: <nil> Dec 27 21:18:09.077260103 (533h49m8.948343148s) [seg_pos={0 0}, upload=468173]: Segment reached optimal threshold of 13 pieces. Dec 27 21:18:09.077474912 (533h49m8.948557947s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077481675 (533h49m8.94856472s) [noise=true , address=77.90.31.21:4878 , piece_id=ZUEU3MRGIZK32C72AZIDFSX7AKZBYYYR7C4SUI3QHNILLRW2IQNA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077637362 (533h49m8.948720397s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077646509 (533h49m8.948729544s) [noise=true , address=45.91.116.14:4191 , piece_id=JEHBVKIIM3PZSPVWAH6HJZVYFH45ACYY6RONJSJ2AVFFQJKWNEQA, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077675063 (533h49m8.948758098s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 27 21:18:09.077681806 (533h49m8.948764841s) [noise=true , address=145.79.227.7:4293 , piece_id=VPHJBJZ5ONJFAASAQQXCEMTIVBAVTSGECW6EU6UZ42UMJLSYGS3A, seg_pos={0 0}, upload=468173]: Long tail cancellation due to optimal threshold Dec 27 21:18:09.077698006 (533h49m8.948781041s) [seg_pos={0 0}, upload=468173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:18:09.077764493 (533h49m8.948847538s) [upload=468173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:18:09.077908978 (533h49m8.948992053s) [upload=468173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:18:09.370129449 (533h49m9.241212494s) [upload=468173]: Done uploading segments... Dec 27 21:18:09.370156701 (533h49m9.241239746s) [upload=468173]: Upload finished. err: <nil> Duration: 30.003375618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X............X....X..XX..X....................X..........] duration: 6.257742369s trace: https://jaeger.dev.storj.io/trace/1c64308627b80a0 FAIL (2025-12-27 21:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ) context deadline exceeded ======LOGS====== 21:08:56.511742125 (533h39m56.38282516s) [upload=468042]: Starting upload Dec 27 21:08:56.511801218 (533h39m56.382884263s) [upload=468042]: Done starting upload Dec 27 21:08:56.511895778 (533h39m56.382978823s) [upload=468042]: Uploading segments... Dec 27 21:08:56.511996580 (533h39m56.383079615s) [upload=468042]: Got next segment. Inline: false Dec 27 21:08:56.512041135 (533h39m56.38312417s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:08:56.512045613 (533h39m56.383128648s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:08:56.763912351 (533h39m56.634995386s) [seg_pos={0 0}, upload=468042]: Begin upload segment... Dec 27 21:08:56.763954471 (533h39m56.635037506s) [seg_pos={0 0}, upload=468042]: Done begin upload segment. Dec 27 21:08:56.763977626 (533h39m56.635060661s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764132270 (533h39m56.635215305s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764647641 (533h39m56.635730676s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764884222 (533h39m56.635967257s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764928115 (533h39m56.63601116s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765181638 (533h39m56.636264673s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765428187 (533h39m56.636511232s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765440381 (533h39m56.636523416s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765702620 (533h39m56.636785665s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766020044 (533h39m56.637103079s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766038740 (533h39m56.637121785s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766616892 (533h39m56.637699947s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767227955 (533h39m56.63831099s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767533326 (533h39m56.638616371s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767908130 (533h39m56.638991175s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.768587243 (533h39m56.639670288s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:58.469544035 (533h39m58.34062707s) [upload=468042]: Next returned nil segment Dec 27 21:08:58.469559786 (533h39m58.340642831s) [upload=468042]: Waiting for error group managing segments... Dec 27 21:08:58.757116725 (533h39m58.62819976s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.764675779 (533h39m58.635758824s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.789933174 (533h39m58.661016219s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.790917848 (533h39m58.662000884s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.791927421 (533h39m58.663010456s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.930859231 (533h39m58.801942266s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.969480152 (533h39m58.840563187s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.995117090 (533h39m58.866200125s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.008463864 (533h39m58.879546899s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.011269727 (533h39m58.882352763s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.024514547 (533h39m58.895597593s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.056588452 (533h39m58.927671497s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094359084 (533h39m58.965442119s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094389091 (533h39m58.965472126s) [seg_pos={0 0}, upload=468042]: Segment reached optimal threshold of 13 pieces. Dec 27 21:08:59.094573222 (533h39m58.965656257s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection Dec 27 21:08:59.094579774 (533h39m58.965662809s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094813589 (533h39m58.965896625s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094819831 (533h39m58.965902866s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094925292 (533h39m58.966008347s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094946863 (533h39m58.966029908s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094978143 (533h39m58.966061198s) [seg_pos={0 0}, upload=468042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:08:59.095101197 (533h39m58.966184242s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:08:59.095111647 (533h39m58.966194683s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:08:59.380206798 (533h39m59.251289833s) [upload=468042]: Done uploading segments... Dec 27 21:08:59.380223310 (533h39m59.251306345s) [upload=468042]: Upload finished. err: <nil> Duration: 6.257771145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX..X............X....X..XX..X....................X.........] duration: 6.029474776s trace: https://jaeger.dev.storj.io/trace/1c64308627b80a0 FAIL (2025-12-27 21:17:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ) context deadline exceeded ======LOGS====== 21:08:56.511742125 (533h39m56.38282516s) [upload=468042]: Starting upload Dec 27 21:08:56.511801218 (533h39m56.382884263s) [upload=468042]: Done starting upload Dec 27 21:08:56.511895778 (533h39m56.382978823s) [upload=468042]: Uploading segments... Dec 27 21:08:56.511996580 (533h39m56.383079615s) [upload=468042]: Got next segment. Inline: false Dec 27 21:08:56.512041135 (533h39m56.38312417s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:08:56.512045613 (533h39m56.383128648s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:08:56.763912351 (533h39m56.634995386s) [seg_pos={0 0}, upload=468042]: Begin upload segment... Dec 27 21:08:56.763954471 (533h39m56.635037506s) [seg_pos={0 0}, upload=468042]: Done begin upload segment. Dec 27 21:08:56.763977626 (533h39m56.635060661s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764132270 (533h39m56.635215305s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764647641 (533h39m56.635730676s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764884222 (533h39m56.635967257s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764928115 (533h39m56.63601116s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765181638 (533h39m56.636264673s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765428187 (533h39m56.636511232s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765440381 (533h39m56.636523416s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765702620 (533h39m56.636785665s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766020044 (533h39m56.637103079s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766038740 (533h39m56.637121785s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766616892 (533h39m56.637699947s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767227955 (533h39m56.63831099s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767533326 (533h39m56.638616371s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767908130 (533h39m56.638991175s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.768587243 (533h39m56.639670288s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:58.469544035 (533h39m58.34062707s) [upload=468042]: Next returned nil segment Dec 27 21:08:58.469559786 (533h39m58.340642831s) [upload=468042]: Waiting for error group managing segments... Dec 27 21:08:58.757116725 (533h39m58.62819976s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.764675779 (533h39m58.635758824s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.789933174 (533h39m58.661016219s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.790917848 (533h39m58.662000884s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.791927421 (533h39m58.663010456s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.930859231 (533h39m58.801942266s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.969480152 (533h39m58.840563187s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.995117090 (533h39m58.866200125s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.008463864 (533h39m58.879546899s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.011269727 (533h39m58.882352763s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.024514547 (533h39m58.895597593s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.056588452 (533h39m58.927671497s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094359084 (533h39m58.965442119s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094389091 (533h39m58.965472126s) [seg_pos={0 0}, upload=468042]: Segment reached optimal threshold of 13 pieces. Dec 27 21:08:59.094573222 (533h39m58.965656257s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection Dec 27 21:08:59.094579774 (533h39m58.965662809s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094813589 (533h39m58.965896625s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094819831 (533h39m58.965902866s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094925292 (533h39m58.966008347s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094946863 (533h39m58.966029908s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094978143 (533h39m58.966061198s) [seg_pos={0 0}, upload=468042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:08:59.095101197 (533h39m58.966184242s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:08:59.095111647 (533h39m58.966194683s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:08:59.380206798 (533h39m59.251289833s) [upload=468042]: Done uploading segments... Dec 27 21:08:59.380223310 (533h39m59.251306345s) [upload=468042]: Upload finished. err: <nil> Duration: 6.029502388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX..X............X....X..XX..X....................X........] duration: 6.404299047s trace: https://jaeger.dev.storj.io/trace/1c64308627b80a0 FAIL (2025-12-27 21:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ) context deadline exceeded ======LOGS====== 21:08:56.511742125 (533h39m56.38282516s) [upload=468042]: Starting upload Dec 27 21:08:56.511801218 (533h39m56.382884263s) [upload=468042]: Done starting upload Dec 27 21:08:56.511895778 (533h39m56.382978823s) [upload=468042]: Uploading segments... Dec 27 21:08:56.511996580 (533h39m56.383079615s) [upload=468042]: Got next segment. Inline: false Dec 27 21:08:56.512041135 (533h39m56.38312417s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:08:56.512045613 (533h39m56.383128648s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:08:56.763912351 (533h39m56.634995386s) [seg_pos={0 0}, upload=468042]: Begin upload segment... Dec 27 21:08:56.763954471 (533h39m56.635037506s) [seg_pos={0 0}, upload=468042]: Done begin upload segment. Dec 27 21:08:56.763977626 (533h39m56.635060661s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764132270 (533h39m56.635215305s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764647641 (533h39m56.635730676s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764884222 (533h39m56.635967257s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764928115 (533h39m56.63601116s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765181638 (533h39m56.636264673s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765428187 (533h39m56.636511232s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765440381 (533h39m56.636523416s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765702620 (533h39m56.636785665s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766020044 (533h39m56.637103079s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766038740 (533h39m56.637121785s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766616892 (533h39m56.637699947s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767227955 (533h39m56.63831099s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767533326 (533h39m56.638616371s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767908130 (533h39m56.638991175s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.768587243 (533h39m56.639670288s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:58.469544035 (533h39m58.34062707s) [upload=468042]: Next returned nil segment Dec 27 21:08:58.469559786 (533h39m58.340642831s) [upload=468042]: Waiting for error group managing segments... Dec 27 21:08:58.757116725 (533h39m58.62819976s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.764675779 (533h39m58.635758824s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.789933174 (533h39m58.661016219s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.790917848 (533h39m58.662000884s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.791927421 (533h39m58.663010456s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.930859231 (533h39m58.801942266s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.969480152 (533h39m58.840563187s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.995117090 (533h39m58.866200125s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.008463864 (533h39m58.879546899s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.011269727 (533h39m58.882352763s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.024514547 (533h39m58.895597593s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.056588452 (533h39m58.927671497s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094359084 (533h39m58.965442119s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094389091 (533h39m58.965472126s) [seg_pos={0 0}, upload=468042]: Segment reached optimal threshold of 13 pieces. Dec 27 21:08:59.094573222 (533h39m58.965656257s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection Dec 27 21:08:59.094579774 (533h39m58.965662809s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094813589 (533h39m58.965896625s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094819831 (533h39m58.965902866s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094925292 (533h39m58.966008347s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094946863 (533h39m58.966029908s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094978143 (533h39m58.966061198s) [seg_pos={0 0}, upload=468042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:08:59.095101197 (533h39m58.966184242s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:08:59.095111647 (533h39m58.966194683s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:08:59.380206798 (533h39m59.251289833s) [upload=468042]: Done uploading segments... Dec 27 21:08:59.380223310 (533h39m59.251306345s) [upload=468042]: Upload finished. err: <nil> Duration: 6.404328692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX..X............X....X..XX..X....................X.......] duration: 28.751701931s trace: https://jaeger.dev.storj.io/trace/1c64308627b80a0 FAIL (2025-12-27 21:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ) context deadline exceeded ======LOGS====== 21:08:56.511742125 (533h39m56.38282516s) [upload=468042]: Starting upload Dec 27 21:08:56.511801218 (533h39m56.382884263s) [upload=468042]: Done starting upload Dec 27 21:08:56.511895778 (533h39m56.382978823s) [upload=468042]: Uploading segments... Dec 27 21:08:56.511996580 (533h39m56.383079615s) [upload=468042]: Got next segment. Inline: false Dec 27 21:08:56.512041135 (533h39m56.38312417s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:08:56.512045613 (533h39m56.383128648s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:08:56.763912351 (533h39m56.634995386s) [seg_pos={0 0}, upload=468042]: Begin upload segment... Dec 27 21:08:56.763954471 (533h39m56.635037506s) [seg_pos={0 0}, upload=468042]: Done begin upload segment. Dec 27 21:08:56.763977626 (533h39m56.635060661s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764132270 (533h39m56.635215305s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764647641 (533h39m56.635730676s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764884222 (533h39m56.635967257s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764928115 (533h39m56.63601116s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765181638 (533h39m56.636264673s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765428187 (533h39m56.636511232s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765440381 (533h39m56.636523416s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765702620 (533h39m56.636785665s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766020044 (533h39m56.637103079s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766038740 (533h39m56.637121785s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766616892 (533h39m56.637699947s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767227955 (533h39m56.63831099s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767533326 (533h39m56.638616371s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767908130 (533h39m56.638991175s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.768587243 (533h39m56.639670288s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:58.469544035 (533h39m58.34062707s) [upload=468042]: Next returned nil segment Dec 27 21:08:58.469559786 (533h39m58.340642831s) [upload=468042]: Waiting for error group managing segments... Dec 27 21:08:58.757116725 (533h39m58.62819976s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.764675779 (533h39m58.635758824s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.789933174 (533h39m58.661016219s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.790917848 (533h39m58.662000884s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.791927421 (533h39m58.663010456s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.930859231 (533h39m58.801942266s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.969480152 (533h39m58.840563187s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.995117090 (533h39m58.866200125s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.008463864 (533h39m58.879546899s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.011269727 (533h39m58.882352763s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.024514547 (533h39m58.895597593s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.056588452 (533h39m58.927671497s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094359084 (533h39m58.965442119s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094389091 (533h39m58.965472126s) [seg_pos={0 0}, upload=468042]: Segment reached optimal threshold of 13 pieces. Dec 27 21:08:59.094573222 (533h39m58.965656257s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection Dec 27 21:08:59.094579774 (533h39m58.965662809s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094813589 (533h39m58.965896625s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094819831 (533h39m58.965902866s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094925292 (533h39m58.966008347s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094946863 (533h39m58.966029908s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094978143 (533h39m58.966061198s) [seg_pos={0 0}, upload=468042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:08:59.095101197 (533h39m58.966184242s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:08:59.095111647 (533h39m58.966194683s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:08:59.380206798 (533h39m59.251289833s) [upload=468042]: Done uploading segments... Dec 27 21:08:59.380223310 (533h39m59.251306345s) [upload=468042]: Upload finished. err: <nil> Duration: 28.751736014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX..X............X....X..XX..X....................X......] duration: 6.107096692s trace: https://jaeger.dev.storj.io/trace/1c64308627b80a0 FAIL (2025-12-27 21:14:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ) context deadline exceeded ======LOGS====== 21:08:56.511742125 (533h39m56.38282516s) [upload=468042]: Starting upload Dec 27 21:08:56.511801218 (533h39m56.382884263s) [upload=468042]: Done starting upload Dec 27 21:08:56.511895778 (533h39m56.382978823s) [upload=468042]: Uploading segments... Dec 27 21:08:56.511996580 (533h39m56.383079615s) [upload=468042]: Got next segment. Inline: false Dec 27 21:08:56.512041135 (533h39m56.38312417s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:08:56.512045613 (533h39m56.383128648s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:08:56.763912351 (533h39m56.634995386s) [seg_pos={0 0}, upload=468042]: Begin upload segment... Dec 27 21:08:56.763954471 (533h39m56.635037506s) [seg_pos={0 0}, upload=468042]: Done begin upload segment. Dec 27 21:08:56.763977626 (533h39m56.635060661s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764132270 (533h39m56.635215305s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764647641 (533h39m56.635730676s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764884222 (533h39m56.635967257s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764928115 (533h39m56.63601116s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765181638 (533h39m56.636264673s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765428187 (533h39m56.636511232s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765440381 (533h39m56.636523416s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765702620 (533h39m56.636785665s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766020044 (533h39m56.637103079s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766038740 (533h39m56.637121785s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766616892 (533h39m56.637699947s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767227955 (533h39m56.63831099s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767533326 (533h39m56.638616371s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767908130 (533h39m56.638991175s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.768587243 (533h39m56.639670288s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:58.469544035 (533h39m58.34062707s) [upload=468042]: Next returned nil segment Dec 27 21:08:58.469559786 (533h39m58.340642831s) [upload=468042]: Waiting for error group managing segments... Dec 27 21:08:58.757116725 (533h39m58.62819976s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.764675779 (533h39m58.635758824s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.789933174 (533h39m58.661016219s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.790917848 (533h39m58.662000884s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.791927421 (533h39m58.663010456s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.930859231 (533h39m58.801942266s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.969480152 (533h39m58.840563187s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.995117090 (533h39m58.866200125s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.008463864 (533h39m58.879546899s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.011269727 (533h39m58.882352763s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.024514547 (533h39m58.895597593s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.056588452 (533h39m58.927671497s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094359084 (533h39m58.965442119s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094389091 (533h39m58.965472126s) [seg_pos={0 0}, upload=468042]: Segment reached optimal threshold of 13 pieces. Dec 27 21:08:59.094573222 (533h39m58.965656257s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection Dec 27 21:08:59.094579774 (533h39m58.965662809s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094813589 (533h39m58.965896625s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094819831 (533h39m58.965902866s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094925292 (533h39m58.966008347s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094946863 (533h39m58.966029908s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094978143 (533h39m58.966061198s) [seg_pos={0 0}, upload=468042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:08:59.095101197 (533h39m58.966184242s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:08:59.095111647 (533h39m58.966194683s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:08:59.380206798 (533h39m59.251289833s) [upload=468042]: Done uploading segments... Dec 27 21:08:59.380223310 (533h39m59.251306345s) [upload=468042]: Upload finished. err: <nil> Duration: 6.107135365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XX..X............X....X..XX..X....................X.....] duration: 6.078048979s trace: https://jaeger.dev.storj.io/trace/1c64308627b80a0 FAIL (2025-12-27 21:13:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ) context deadline exceeded ======LOGS====== 21:08:56.511742125 (533h39m56.38282516s) [upload=468042]: Starting upload Dec 27 21:08:56.511801218 (533h39m56.382884263s) [upload=468042]: Done starting upload Dec 27 21:08:56.511895778 (533h39m56.382978823s) [upload=468042]: Uploading segments... Dec 27 21:08:56.511996580 (533h39m56.383079615s) [upload=468042]: Got next segment. Inline: false Dec 27 21:08:56.512041135 (533h39m56.38312417s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 21:08:56.512045613 (533h39m56.383128648s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 21:08:56.763912351 (533h39m56.634995386s) [seg_pos={0 0}, upload=468042]: Begin upload segment... Dec 27 21:08:56.763954471 (533h39m56.635037506s) [seg_pos={0 0}, upload=468042]: Done begin upload segment. Dec 27 21:08:56.763977626 (533h39m56.635060661s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764132270 (533h39m56.635215305s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764647641 (533h39m56.635730676s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764884222 (533h39m56.635967257s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.764928115 (533h39m56.63601116s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765181638 (533h39m56.636264673s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765428187 (533h39m56.636511232s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765440381 (533h39m56.636523416s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.765702620 (533h39m56.636785665s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766020044 (533h39m56.637103079s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766038740 (533h39m56.637121785s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.766616892 (533h39m56.637699947s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767227955 (533h39m56.63831099s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767533326 (533h39m56.638616371s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.767908130 (533h39m56.638991175s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:56.768587243 (533h39m56.639670288s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Uploading piece... Dec 27 21:08:58.469544035 (533h39m58.34062707s) [upload=468042]: Next returned nil segment Dec 27 21:08:58.469559786 (533h39m58.340642831s) [upload=468042]: Waiting for error group managing segments... Dec 27 21:08:58.757116725 (533h39m58.62819976s) [noise=true , address=145.79.227.25:4619 , piece_id=CYOX2BEHGN46LDRSDAOVICE42BF25KYVFSZSVHKZCWU3TFO55UUA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.764675779 (533h39m58.635758824s) [noise=true , address=145.79.227.45:4229 , piece_id=WU4Y2DOSF6ZTKFWI43AKVCBAGPREEZ4KJACW7ATVK2TZKQHFE7HQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.789933174 (533h39m58.661016219s) [noise=true , address=145.79.227.36:4519 , piece_id=HNAIVQYGBFQEROYZVQHBR62YAYN55GEC466DMCRXFFF4ZERFZYWA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.790917848 (533h39m58.662000884s) [noise=true , address=145.79.227.8:4329 , piece_id=CTCPTXNHZ3TNRDGZCLCUS6FGZX3E6AW7K7QPLCAFA3LBZS3DFOBA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.791927421 (533h39m58.663010456s) [noise=true , address=145.79.227.14:4837 , piece_id=MGRXPRVIK6OSNM4LEIYGGMQLANYCHLZ4JB2LUODSBP65TWHKEITA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.930859231 (533h39m58.801942266s) [noise=true , address=77.90.31.15:4700 , piece_id=ZME2CMC2MTBVKK72RSRT4NDGQ4GZ2UNLYB3CFWHZOEX3MWVCZLGQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.969480152 (533h39m58.840563187s) [noise=true , address=45.91.116.5:4298 , piece_id=XH2634QSUHQKNHZLA25H6XTFCEWTFEBZV4VVVTP4BPZLIBVS3ZMQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:58.995117090 (533h39m58.866200125s) [noise=true , address=45.91.116.8:4046 , piece_id=5RMC5ZBEIAVZZYRKCWMCQS24QI54QLGGIYMJBBWOZA4PNVXKAG2Q, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.008463864 (533h39m58.879546899s) [noise=true , address=45.91.116.7:4012 , piece_id=QYQOCIT5TAGHRYCVITXM75ERPVF5B6X34VJNGUVN24U2UY3FI2UA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.011269727 (533h39m58.882352763s) [noise=true , address=45.91.116.13:4153 , piece_id=VWV7WNHVXZN3ANZY3T6SEPFBOJX67DOO65ON6AFFUFPT5XONHFQQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.024514547 (533h39m58.895597593s) [noise=true , address=45.91.116.9:4261 , piece_id=UKKHZLRI7MJP7SYUQM3ZIU3NEKGHDSFHF3QPIDW6DKIR5T5CCT3A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.056588452 (533h39m58.927671497s) [noise=true , address=77.90.31.18:4808 , piece_id=X5JBJ3DQCJ57XG4WGCRJOPLARUECSHOVO2T4FQB4BR66B33LR57A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094359084 (533h39m58.965442119s) [noise=true , address=45.91.116.10:4084 , piece_id=RZTHUNICY6M6YZXC2UH43N2PUMFXSOQLSHZFJEF4W4RV7FG5PHPQ, seg_pos={0 0}, upload=468042]: Done uploading piece. err: <nil> Dec 27 21:08:59.094389091 (533h39m58.965472126s) [seg_pos={0 0}, upload=468042]: Segment reached optimal threshold of 13 pieces. Dec 27 21:08:59.094573222 (533h39m58.965656257s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44710->145.79.227.47:4265: use of closed network connection Dec 27 21:08:59.094579774 (533h39m58.965662809s) [noise=true , address=145.79.227.47:4265 , piece_id=Y3B2UH6Y5UN6C2TQFYHJ77VWCD5F5VEYVESCV7IXCNNNLADKMS6A, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094813589 (533h39m58.965896625s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094819831 (533h39m58.965902866s) [noise=true , address=77.90.31.10:4517 , piece_id=ELYVXSJVRQ6TQVSDRFIKRNBPN5BKFAMMRHT4SCHGXOZAL3EVU6TA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094925292 (533h39m58.966008347s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 21:08:59.094946863 (533h39m58.966029908s) [noise=true , address=77.90.31.12:4589 , piece_id=GJEX3GWFYGHG7LS7BNRQWE6ZUYSGKINTYPZY6QCHZLN54WOD3SEA, seg_pos={0 0}, upload=468042]: Long tail cancellation due to optimal threshold Dec 27 21:08:59.094978143 (533h39m58.966061198s) [seg_pos={0 0}, upload=468042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 21:08:59.095101197 (533h39m58.966184242s) [upload=468042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 21:08:59.095111647 (533h39m58.966194683s) [upload=468042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 21:08:59.380206798 (533h39m59.251289833s) [upload=468042]: Done uploading segments... Dec 27 21:08:59.380223310 (533h39m59.251306345s) [upload=468042]: Upload finished. err: <nil> Duration: 6.078077123s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)