Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X.] duration: 5.804896469s trace: https://jaeger.dev.storj.io/trace/8672f543629964b FAIL (2025-12-27 11:28:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA) context deadline exceeded ======LOGS====== 27 11:27:39.225373904 (523h58m39.096456939s) [upload=459651]: Starting upload Dec 27 11:27:39.225459026 (523h58m39.096542081s) [upload=459651]: Done starting upload Dec 27 11:27:39.225641684 (523h58m39.096724729s) [upload=459651]: Uploading segments... Dec 27 11:27:39.225780358 (523h58m39.096863403s) [upload=459651]: Got next segment. Inline: false Dec 27 11:27:39.225978806 (523h58m39.097061851s) [upload=459651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:27:39.225986190 (523h58m39.097069225s) [upload=459651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:27:39.470471501 (523h58m39.341554546s) [seg_pos={0 0}, upload=459651]: Begin upload segment... Dec 27 11:27:39.470535303 (523h58m39.341618338s) [seg_pos={0 0}, upload=459651]: Done begin upload segment. Dec 27 11:27:39.470617759 (523h58m39.341700794s) [noise=true , address=145.79.227.29:4942 , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.470638800 (523h58m39.341721835s) [noise=true , address=145.79.227.38:4978 , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.471138992 (523h58m39.342222027s) [noise=true , address=145.79.227.46:4602 , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.471638824 (523h58m39.342721869s) [noise=true , address=145.79.227.37:4567 , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.472129819 (523h58m39.343212854s) [noise=true , address=45.91.116.10:4083 , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.472600655 (523h58m39.3436837s) [noise=true , address=145.79.227.14:4837 , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.473040142 (523h58m39.344123177s) [noise=true , address=45.91.116.9:4263 , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.474708760 (523h58m39.345791795s) [noise=true , address=45.91.116.4:4939 , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.475038137 (523h58m39.346121182s) [noise=true , address=145.79.227.49:4337 , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.477105023 (523h58m39.348188058s) [noise=true , address=145.79.227.23:4546 , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.479241691 (523h58m39.350324736s) [noise=true , address=77.90.31.11:4556 , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.480713945 (523h58m39.35179698s) [noise=true , address=145.79.227.2:4113 , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.481554245 (523h58m39.35263729s) [noise=true , address=145.79.227.42:4123 , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.482950694 (523h58m39.354033729s) [noise=true , address=145.79.227.28:4726 , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.483820180 (523h58m39.354903215s) [noise=true , address=145.79.227.20:4438 , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.484378763 (523h58m39.355461798s) [noise=true , address=45.91.116.14:4190 , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:41.195631697 (523h58m41.066714742s) [upload=459651]: Next returned nil segment Dec 27 11:27:41.195649310 (523h58m41.066732355s) [upload=459651]: Waiting for error group managing segments... Dec 27 11:27:41.479723735 (523h58m41.35080677s) [noise=true , address=145.79.227.28:4726 , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.483476681 (523h58m41.354559726s) [noise=true , address=145.79.227.20:4438 , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.483525936 (523h58m41.354608971s) [noise=true , address=145.79.227.29:4942 , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.486140745 (523h58m41.35722379s) [noise=true , address=145.79.227.2:4113 , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.491385140 (523h58m41.362468175s) [noise=true , address=145.79.227.49:4337 , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.503239233 (523h58m41.374322268s) [noise=true , address=145.79.227.46:4602 , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.508356327 (523h58m41.379439362s) [noise=true , address=145.79.227.42:4123 , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.508668511 (523h58m41.379751546s) [noise=true , address=145.79.227.37:4567 , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.509179284 (523h58m41.380262319s) [noise=true , address=145.79.227.38:4978 , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.511729139 (523h58m41.382812184s) [noise=true , address=145.79.227.14:4837 , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.513883672 (523h58m41.384966707s) [noise=true , address=145.79.227.23:4546 , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.652583589 (523h58m41.523666624s) [noise=true , address=77.90.31.11:4556 , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.851675284 (523h58m41.722758319s) [noise=true , address=45.91.116.14:4190 , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.851694841 (523h58m41.722777876s) [seg_pos={0 0}, upload=459651]: Segment reached optimal threshold of 13 pieces. Dec 27 11:27:41.851917645 (523h58m41.72300069s) [noise=true , address=45.91.116.10:4083 , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection Dec 27 11:27:41.851924338 (523h58m41.723007373s) [noise=true , address=45.91.116.10:4083 , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold Dec 27 11:27:41.852110061 (523h58m41.723193096s) [noise=true , address=45.91.116.4:4939 , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 11:27:41.852142303 (523h58m41.723225338s) [noise=true , address=45.91.116.4:4939 , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold Dec 27 11:27:41.852265296 (523h58m41.723348332s) [noise=true , address=45.91.116.9:4263 , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 11:27:41.852271890 (523h58m41.723354925s) [noise=true , address=45.91.116.9:4263 , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold Dec 27 11:27:41.852290224 (523h58m41.723373269s) [seg_pos={0 0}, upload=459651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:27:41.852381198 (523h58m41.723464233s) [upload=459651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:27:41.852386618 (523h58m41.723469653s) [upload=459651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:27:42.145941195 (523h58m42.01702423s) [upload=459651]: Done uploading segments... Dec 27 11:27:42.145963247 (523h58m42.017046292s) [upload=459651]: Upload finished. err: <nil> Duration: 5.804913982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X] duration: 30.004000573s trace: https://jaeger.dev.storj.io/trace/8672f543629964b FAIL (2025-12-27 11:28:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA) context deadline exceeded ======LOGS====== 27 11:27:39.225373904 (523h58m39.096456939s) [upload=459651]: Starting upload Dec 27 11:27:39.225459026 (523h58m39.096542081s) [upload=459651]: Done starting upload Dec 27 11:27:39.225641684 (523h58m39.096724729s) [upload=459651]: Uploading segments... Dec 27 11:27:39.225780358 (523h58m39.096863403s) [upload=459651]: Got next segment. Inline: false Dec 27 11:27:39.225978806 (523h58m39.097061851s) [upload=459651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:27:39.225986190 (523h58m39.097069225s) [upload=459651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:27:39.470471501 (523h58m39.341554546s) [seg_pos={0 0}, upload=459651]: Begin upload segment... Dec 27 11:27:39.470535303 (523h58m39.341618338s) [seg_pos={0 0}, upload=459651]: Done begin upload segment. Dec 27 11:27:39.470617759 (523h58m39.341700794s) [noise=true , address=145.79.227.29:4942 , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.470638800 (523h58m39.341721835s) [noise=true , address=145.79.227.38:4978 , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.471138992 (523h58m39.342222027s) [noise=true , address=145.79.227.46:4602 , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.471638824 (523h58m39.342721869s) [noise=true , address=145.79.227.37:4567 , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.472129819 (523h58m39.343212854s) [noise=true , address=45.91.116.10:4083 , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.472600655 (523h58m39.3436837s) [noise=true , address=145.79.227.14:4837 , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.473040142 (523h58m39.344123177s) [noise=true , address=45.91.116.9:4263 , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.474708760 (523h58m39.345791795s) [noise=true , address=45.91.116.4:4939 , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.475038137 (523h58m39.346121182s) [noise=true , address=145.79.227.49:4337 , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.477105023 (523h58m39.348188058s) [noise=true , address=145.79.227.23:4546 , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.479241691 (523h58m39.350324736s) [noise=true , address=77.90.31.11:4556 , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.480713945 (523h58m39.35179698s) [noise=true , address=145.79.227.2:4113 , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.481554245 (523h58m39.35263729s) [noise=true , address=145.79.227.42:4123 , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.482950694 (523h58m39.354033729s) [noise=true , address=145.79.227.28:4726 , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.483820180 (523h58m39.354903215s) [noise=true , address=145.79.227.20:4438 , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:39.484378763 (523h58m39.355461798s) [noise=true , address=45.91.116.14:4190 , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Uploading piece... Dec 27 11:27:41.195631697 (523h58m41.066714742s) [upload=459651]: Next returned nil segment Dec 27 11:27:41.195649310 (523h58m41.066732355s) [upload=459651]: Waiting for error group managing segments... Dec 27 11:27:41.479723735 (523h58m41.35080677s) [noise=true , address=145.79.227.28:4726 , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.483476681 (523h58m41.354559726s) [noise=true , address=145.79.227.20:4438 , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.483525936 (523h58m41.354608971s) [noise=true , address=145.79.227.29:4942 , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.486140745 (523h58m41.35722379s) [noise=true , address=145.79.227.2:4113 , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.491385140 (523h58m41.362468175s) [noise=true , address=145.79.227.49:4337 , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.503239233 (523h58m41.374322268s) [noise=true , address=145.79.227.46:4602 , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.508356327 (523h58m41.379439362s) [noise=true , address=145.79.227.42:4123 , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.508668511 (523h58m41.379751546s) [noise=true , address=145.79.227.37:4567 , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.509179284 (523h58m41.380262319s) [noise=true , address=145.79.227.38:4978 , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.511729139 (523h58m41.382812184s) [noise=true , address=145.79.227.14:4837 , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.513883672 (523h58m41.384966707s) [noise=true , address=145.79.227.23:4546 , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.652583589 (523h58m41.523666624s) [noise=true , address=77.90.31.11:4556 , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.851675284 (523h58m41.722758319s) [noise=true , address=45.91.116.14:4190 , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil> Dec 27 11:27:41.851694841 (523h58m41.722777876s) [seg_pos={0 0}, upload=459651]: Segment reached optimal threshold of 13 pieces. Dec 27 11:27:41.851917645 (523h58m41.72300069s) [noise=true , address=45.91.116.10:4083 , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection Dec 27 11:27:41.851924338 (523h58m41.723007373s) [noise=true , address=45.91.116.10:4083 , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold Dec 27 11:27:41.852110061 (523h58m41.723193096s) [noise=true , address=45.91.116.4:4939 , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 11:27:41.852142303 (523h58m41.723225338s) [noise=true , address=45.91.116.4:4939 , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold Dec 27 11:27:41.852265296 (523h58m41.723348332s) [noise=true , address=45.91.116.9:4263 , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 11:27:41.852271890 (523h58m41.723354925s) [noise=true , address=45.91.116.9:4263 , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold Dec 27 11:27:41.852290224 (523h58m41.723373269s) [seg_pos={0 0}, upload=459651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:27:41.852381198 (523h58m41.723464233s) [upload=459651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:27:41.852386618 (523h58m41.723469653s) [upload=459651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:27:42.145941195 (523h58m42.01702423s) [upload=459651]: Done uploading segments... Dec 27 11:27:42.145963247 (523h58m42.017046292s) [upload=459651]: Upload finished. err: <nil> Duration: 30.004021883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.] duration: 5.911418197s trace: https://jaeger.dev.storj.io/trace/41a1a192ea70a2ab FAIL (2025-12-27 11:27:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 11:26:14.991870153 (523h57m14.862953188s) [upload=459629]: Starting upload Dec 27 11:26:14.991948872 (523h57m14.863031907s) [upload=459629]: Done starting upload Dec 27 11:26:14.992071717 (523h57m14.863154752s) [upload=459629]: Uploading segments... Dec 27 11:26:14.992146669 (523h57m14.863229704s) [upload=459629]: Got next segment. Inline: false Dec 27 11:26:14.992178159 (523h57m14.863261194s) [upload=459629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:26:14.992183569 (523h57m14.863266614s) [upload=459629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:26:15.235614142 (523h57m15.106697197s) [seg_pos={0 0}, upload=459629]: Begin upload segment... Dec 27 11:26:15.235670911 (523h57m15.106753956s) [seg_pos={0 0}, upload=459629]: Done begin upload segment. Dec 27 11:26:15.235706258 (523h57m15.106789303s) [noise=true , address=145.79.227.29:4942 , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.235778025 (523h57m15.10686107s) [noise=true , address=145.79.227.7:4294 , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.236323093 (523h57m15.107406138s) [noise=true , address=145.79.227.20:4438 , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.236734055 (523h57m15.10781709s) [noise=true , address=45.91.116.8:4047 , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.237171578 (523h57m15.108254613s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.237612979 (523h57m15.108696014s) [noise=true , address=145.79.227.5:4221 , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.238197301 (523h57m15.109280346s) [noise=true , address=145.79.227.17:4971 , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.238637018 (523h57m15.109720053s) [noise=true , address=145.79.227.2:4113 , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.239459916 (523h57m15.110542951s) [noise=true , address=45.91.116.11:4119 , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.240239019 (523h57m15.111322064s) [noise=true , address=45.91.116.6:4976 , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.240618111 (523h57m15.111701146s) [noise=true , address=77.90.31.16:4736 , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.241668441 (523h57m15.112751476s) [noise=true , address=45.91.116.12:4228 , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.242852726 (523h57m15.113935761s) [noise=true , address=145.79.227.48:4303 , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.243884661 (523h57m15.114967696s) [noise=true , address=145.79.227.42:4122 , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.245598725 (523h57m15.11668177s) [noise=true , address=145.79.227.50:4373 , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.245979209 (523h57m15.117062254s) [noise=true , address=145.79.227.28:4726 , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:16.958366904 (523h57m16.829449939s) [upload=459629]: Next returned nil segment Dec 27 11:26:16.958382122 (523h57m16.829465157s) [upload=459629]: Waiting for error group managing segments... Dec 27 11:26:17.233391617 (523h57m17.104474662s) [noise=true , address=145.79.227.48:4303 , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.249989252 (523h57m17.121072287s) [noise=true , address=145.79.227.17:4971 , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.253046353 (523h57m17.124129388s) [noise=true , address=145.79.227.29:4942 , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.253447597 (523h57m17.124530642s) [noise=true , address=145.79.227.20:4438 , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.254014657 (523h57m17.125097692s) [noise=true , address=145.79.227.42:4122 , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.254441289 (523h57m17.125524324s) [noise=true , address=145.79.227.28:4726 , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.262470919 (523h57m17.133553954s) [noise=true , address=145.79.227.2:4113 , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.267264416 (523h57m17.138347451s) [noise=true , address=145.79.227.5:4221 , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.269124297 (523h57m17.140207343s) [noise=true , address=145.79.227.7:4294 , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.275073265 (523h57m17.1461563s) [noise=true , address=145.79.227.50:4373 , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.428639562 (523h57m17.299722607s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.483768196 (523h57m17.354851232s) [noise=true , address=45.91.116.6:4976 , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.495914796 (523h57m17.366997831s) [noise=true , address=45.91.116.12:4228 , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.495934413 (523h57m17.367017458s) [seg_pos={0 0}, upload=459629]: Segment reached optimal threshold of 13 pieces. Dec 27 11:26:17.496159161 (523h57m17.367242206s) [noise=true , address=45.91.116.8:4047 , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:26:17.496167376 (523h57m17.367250411s) [noise=true , address=45.91.116.8:4047 , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold Dec 27 11:26:17.496189268 (523h57m17.367272314s) [noise=true , address=77.90.31.16:4736 , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 11:26:17.496210999 (523h57m17.367294034s) [noise=true , address=77.90.31.16:4736 , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold Dec 27 11:26:17.496375122 (523h57m17.367458177s) [noise=true , address=45.91.116.11:4119 , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 11:26:17.496381043 (523h57m17.367464078s) [noise=true , address=45.91.116.11:4119 , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold Dec 27 11:26:17.496403096 (523h57m17.367486131s) [seg_pos={0 0}, upload=459629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:26:17.496465124 (523h57m17.367548169s) [upload=459629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:26:17.496476074 (523h57m17.367559109s) [upload=459629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:26:17.783203010 (523h57m17.654286075s) [upload=459629]: Done uploading segments... Dec 27 11:26:17.783221195 (523h57m17.65430424s) [upload=459629]: Upload finished. err: <nil> Duration: 5.911453795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X] duration: 30.003176243s trace: https://jaeger.dev.storj.io/trace/41a1a192ea70a2ab FAIL (2025-12-27 11:26:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 11:26:14.991870153 (523h57m14.862953188s) [upload=459629]: Starting upload Dec 27 11:26:14.991948872 (523h57m14.863031907s) [upload=459629]: Done starting upload Dec 27 11:26:14.992071717 (523h57m14.863154752s) [upload=459629]: Uploading segments... Dec 27 11:26:14.992146669 (523h57m14.863229704s) [upload=459629]: Got next segment. Inline: false Dec 27 11:26:14.992178159 (523h57m14.863261194s) [upload=459629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:26:14.992183569 (523h57m14.863266614s) [upload=459629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:26:15.235614142 (523h57m15.106697197s) [seg_pos={0 0}, upload=459629]: Begin upload segment... Dec 27 11:26:15.235670911 (523h57m15.106753956s) [seg_pos={0 0}, upload=459629]: Done begin upload segment. Dec 27 11:26:15.235706258 (523h57m15.106789303s) [noise=true , address=145.79.227.29:4942 , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.235778025 (523h57m15.10686107s) [noise=true , address=145.79.227.7:4294 , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.236323093 (523h57m15.107406138s) [noise=true , address=145.79.227.20:4438 , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.236734055 (523h57m15.10781709s) [noise=true , address=45.91.116.8:4047 , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.237171578 (523h57m15.108254613s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.237612979 (523h57m15.108696014s) [noise=true , address=145.79.227.5:4221 , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.238197301 (523h57m15.109280346s) [noise=true , address=145.79.227.17:4971 , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.238637018 (523h57m15.109720053s) [noise=true , address=145.79.227.2:4113 , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.239459916 (523h57m15.110542951s) [noise=true , address=45.91.116.11:4119 , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.240239019 (523h57m15.111322064s) [noise=true , address=45.91.116.6:4976 , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.240618111 (523h57m15.111701146s) [noise=true , address=77.90.31.16:4736 , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.241668441 (523h57m15.112751476s) [noise=true , address=45.91.116.12:4228 , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.242852726 (523h57m15.113935761s) [noise=true , address=145.79.227.48:4303 , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.243884661 (523h57m15.114967696s) [noise=true , address=145.79.227.42:4122 , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.245598725 (523h57m15.11668177s) [noise=true , address=145.79.227.50:4373 , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:15.245979209 (523h57m15.117062254s) [noise=true , address=145.79.227.28:4726 , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Uploading piece... Dec 27 11:26:16.958366904 (523h57m16.829449939s) [upload=459629]: Next returned nil segment Dec 27 11:26:16.958382122 (523h57m16.829465157s) [upload=459629]: Waiting for error group managing segments... Dec 27 11:26:17.233391617 (523h57m17.104474662s) [noise=true , address=145.79.227.48:4303 , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.249989252 (523h57m17.121072287s) [noise=true , address=145.79.227.17:4971 , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.253046353 (523h57m17.124129388s) [noise=true , address=145.79.227.29:4942 , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.253447597 (523h57m17.124530642s) [noise=true , address=145.79.227.20:4438 , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.254014657 (523h57m17.125097692s) [noise=true , address=145.79.227.42:4122 , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.254441289 (523h57m17.125524324s) [noise=true , address=145.79.227.28:4726 , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.262470919 (523h57m17.133553954s) [noise=true , address=145.79.227.2:4113 , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.267264416 (523h57m17.138347451s) [noise=true , address=145.79.227.5:4221 , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.269124297 (523h57m17.140207343s) [noise=true , address=145.79.227.7:4294 , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.275073265 (523h57m17.1461563s) [noise=true , address=145.79.227.50:4373 , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.428639562 (523h57m17.299722607s) [noise=true , address=77.90.31.11:4555 , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.483768196 (523h57m17.354851232s) [noise=true , address=45.91.116.6:4976 , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.495914796 (523h57m17.366997831s) [noise=true , address=45.91.116.12:4228 , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil> Dec 27 11:26:17.495934413 (523h57m17.367017458s) [seg_pos={0 0}, upload=459629]: Segment reached optimal threshold of 13 pieces. Dec 27 11:26:17.496159161 (523h57m17.367242206s) [noise=true , address=45.91.116.8:4047 , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:26:17.496167376 (523h57m17.367250411s) [noise=true , address=45.91.116.8:4047 , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold Dec 27 11:26:17.496189268 (523h57m17.367272314s) [noise=true , address=77.90.31.16:4736 , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 11:26:17.496210999 (523h57m17.367294034s) [noise=true , address=77.90.31.16:4736 , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold Dec 27 11:26:17.496375122 (523h57m17.367458177s) [noise=true , address=45.91.116.11:4119 , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 11:26:17.496381043 (523h57m17.367464078s) [noise=true , address=45.91.116.11:4119 , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold Dec 27 11:26:17.496403096 (523h57m17.367486131s) [seg_pos={0 0}, upload=459629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:26:17.496465124 (523h57m17.367548169s) [upload=459629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:26:17.496476074 (523h57m17.367559109s) [upload=459629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:26:17.783203010 (523h57m17.654286075s) [upload=459629]: Done uploading segments... Dec 27 11:26:17.783221195 (523h57m17.65430424s) [upload=459629]: Upload finished. err: <nil> Duration: 30.00320031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.] duration: 5.594744817s trace: https://jaeger.dev.storj.io/trace/42d146cbf8df32b5 FAIL (2025-12-27 11:25:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 11:25:06.167671957 (523h56m6.038755002s) [upload=459611]: Starting upload Dec 27 11:25:06.167757741 (523h56m6.038840776s) [upload=459611]: Done starting upload Dec 27 11:25:06.167863993 (523h56m6.038947048s) [upload=459611]: Uploading segments... Dec 27 11:25:06.168079964 (523h56m6.039163009s) [upload=459611]: Got next segment. Inline: false Dec 27 11:25:06.168145319 (523h56m6.039228364s) [upload=459611]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:25:06.168154135 (523h56m6.03923718s) [upload=459611]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:25:06.421674230 (523h56m6.292757275s) [seg_pos={0 0}, upload=459611]: Begin upload segment... Dec 27 11:25:06.421738753 (523h56m6.292821798s) [seg_pos={0 0}, upload=459611]: Done begin upload segment. Dec 27 11:25:06.421916251 (523h56m6.292999296s) [noise=true , address=145.79.227.50:4375 , piece_id=4NXNAYZZUHVPIAXI54U4WVUWZJQCE3VIDQ7SUXJ6CBQJ7ZQAHJFA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.421980444 (523h56m6.293063479s) [noise=true , address=145.79.227.46:4602 , piece_id=DYDB6H75QHHQNI5A75DUCFNC4JM5E5IYEV4GP34MZWXD6FWALOVA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.422671820 (523h56m6.293754865s) [noise=true , address=77.90.31.21:4878 , piece_id=LNS6AMLEKQEIF5JEN6UACQVFSGRG6HAS5W4MBOJAW6KDYQO5CM7A, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.422926996 (523h56m6.294010041s) [noise=true , address=45.91.116.11:4120 , piece_id=K5RLQYJBNOKM27GOBVGDIV7O4IRXMHMMEKBP7YYFE62NG6BO5CSQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.423461373 (523h56m6.294544418s) [noise=true , address=145.79.227.42:4123 , piece_id=53XNDYY5RLAQIF5HR4FBBLHXZTIJU65QN52FOAYXVCZT2RSCCMNA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.423864100 (523h56m6.294947145s) [noise=true , address=145.79.227.5:4221 , piece_id=NVQZU2Z7FOGABIX6ALMXCQIQBFYDSA2DEYDHK5TNPOPLXN7ARK6Q, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.424272348 (523h56m6.295355393s) [noise=true , address=77.90.31.14:4661 , piece_id=T77SHWDIOKJ2ULPQ577XDWZSQZ6RHTDSK7IVYUSIHZDZEMR3OGXQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.424914701 (523h56m6.295997746s) [noise=true , address=145.79.227.20:4438 , piece_id=DGGSSDMZN6GPWT7E45MXDLN2HIH5WOANOZO6PYP4EPGIWKH3XB4Q, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.425344961 (523h56m6.296428006s) [noise=true , address=145.79.227.2:4113 , piece_id=QK6SBPDK6QGMG5USMQBF4LLKU6IKTZ2J2LRONDQX4X2MY6I63IPA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.425847427 (523h56m6.296930482s) [noise=true , address=45.91.116.6:4975 , piece_id=D4MDIFGGVQ6GXCBD3EE4FC4XEEEI6XT2TSDJV4VX6G62TZA4SENA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.430273525 (523h56m6.30135656s) [noise=true , address=145.79.227.4:4186 , piece_id=GHDLMEG4UOKIHNBA3NZY7XBAMUZD5A3KD6QUUS2V4SWQDJE2K2LA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.430866284 (523h56m6.301949329s) [noise=true , address=77.90.31.10:4517 , piece_id=F7JVENE5H6CIRARN5LLG4VKLGMKB6ZRDBHKCJXSUOP6Z26FYZILA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.431369062 (523h56m6.302452107s) [noise=true , address=145.79.227.13:4802 , piece_id=Z75VG647RDW43724SKOQVE7J6EYMXI634R2SIS36DZSECVMMU7FQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.431871428 (523h56m6.302954473s) [noise=true , address=145.79.227.14:4837 , piece_id=4AHK2EPGMNXRCFR3TUEMAM2ZXE3ZAI6DYDGLTXB553QI46FOTJWA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.432809594 (523h56m6.303892629s) [noise=true , address=145.79.227.38:4978 , piece_id=UDAGYOMY3WVVS6SUXKIUPXUEHC5E2KMV3TDYTXJT33E25WTQBIZA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.435090277 (523h56m6.306173323s) [noise=true , address=145.79.227.41:4085 , piece_id=UYLG3PUNGDR6IKEKB43YWITHSCDA5JV2ND5FLC26MKHBRBYMAFVQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:08.047099283 (523h56m7.918182328s) [upload=459611]: Next returned nil segment Dec 27 11:25:08.047112739 (523h56m7.918195774s) [upload=459611]: Waiting for error group managing segments... Dec 27 11:25:08.430945751 (523h56m8.302028796s) [noise=true , address=145.79.227.14:4837 , piece_id=4AHK2EPGMNXRCFR3TUEMAM2ZXE3ZAI6DYDGLTXB553QI46FOTJWA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.431725796 (523h56m8.302808831s) [noise=true , address=145.79.227.38:4978 , piece_id=UDAGYOMY3WVVS6SUXKIUPXUEHC5E2KMV3TDYTXJT33E25WTQBIZA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.435841935 (523h56m8.30692497s) [noise=true , address=145.79.227.20:4438 , piece_id=DGGSSDMZN6GPWT7E45MXDLN2HIH5WOANOZO6PYP4EPGIWKH3XB4Q, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.436153698 (523h56m8.307236733s) [noise=true , address=145.79.227.2:4113 , piece_id=QK6SBPDK6QGMG5USMQBF4LLKU6IKTZ2J2LRONDQX4X2MY6I63IPA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.443356844 (523h56m8.314439879s) [noise=true , address=145.79.227.50:4375 , piece_id=4NXNAYZZUHVPIAXI54U4WVUWZJQCE3VIDQ7SUXJ6CBQJ7ZQAHJFA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.446780423 (523h56m8.317863459s) [noise=true , address=145.79.227.41:4085 , piece_id=UYLG3PUNGDR6IKEKB43YWITHSCDA5JV2ND5FLC26MKHBRBYMAFVQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.449574013 (523h56m8.320657058s) [noise=true , address=145.79.227.5:4221 , piece_id=NVQZU2Z7FOGABIX6ALMXCQIQBFYDSA2DEYDHK5TNPOPLXN7ARK6Q, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.453222270 (523h56m8.324305315s) [noise=true , address=145.79.227.46:4602 , piece_id=DYDB6H75QHHQNI5A75DUCFNC4JM5E5IYEV4GP34MZWXD6FWALOVA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.459121232 (523h56m8.330204267s) [noise=true , address=145.79.227.4:4186 , piece_id=GHDLMEG4UOKIHNBA3NZY7XBAMUZD5A3KD6QUUS2V4SWQDJE2K2LA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.460885442 (523h56m8.331968477s) [noise=true , address=145.79.227.42:4123 , piece_id=53XNDYY5RLAQIF5HR4FBBLHXZTIJU65QN52FOAYXVCZT2RSCCMNA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.466587089 (523h56m8.337670134s) [noise=true , address=145.79.227.13:4802 , piece_id=Z75VG647RDW43724SKOQVE7J6EYMXI634R2SIS36DZSECVMMU7FQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.611439863 (523h56m8.482522898s) [noise=true , address=77.90.31.14:4661 , piece_id=T77SHWDIOKJ2ULPQ577XDWZSQZ6RHTDSK7IVYUSIHZDZEMR3OGXQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.659568017 (523h56m8.530651062s) [noise=true , address=77.90.31.21:4878 , piece_id=LNS6AMLEKQEIF5JEN6UACQVFSGRG6HAS5W4MBOJAW6KDYQO5CM7A, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.659592274 (523h56m8.530675309s) [seg_pos={0 0}, upload=459611]: Segment reached optimal threshold of 13 pieces. Dec 27 11:25:08.659804528 (523h56m8.530887573s) [noise=true , address=77.90.31.10:4517 , piece_id=F7JVENE5H6CIRARN5LLG4VKLGMKB6ZRDBHKCJXSUOP6Z26FYZILA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 11:25:08.659814256 (523h56m8.530897301s) [noise=true , address=77.90.31.10:4517 , piece_id=F7JVENE5H6CIRARN5LLG4VKLGMKB6ZRDBHKCJXSUOP6Z26FYZILA, seg_pos={0 0}, upload=459611]: Long tail cancellation due to optimal threshold Dec 27 11:25:08.660005109 (523h56m8.531088154s) [noise=true , address=45.91.116.11:4120 , piece_id=K5RLQYJBNOKM27GOBVGDIV7O4IRXMHMMEKBP7YYFE62NG6BO5CSQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 11:25:08.660013415 (523h56m8.53109645s) [noise=true , address=45.91.116.11:4120 , piece_id=K5RLQYJBNOKM27GOBVGDIV7O4IRXMHMMEKBP7YYFE62NG6BO5CSQ, seg_pos={0 0}, upload=459611]: Long tail cancellation due to optimal threshold Dec 27 11:25:08.660296905 (523h56m8.53137994s) [noise=true , address=45.91.116.6:4975 , piece_id=D4MDIFGGVQ6GXCBD3EE4FC4XEEEI6XT2TSDJV4VX6G62TZA4SENA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:32912->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 11:25:08.660305922 (523h56m8.531388967s) [noise=true , address=45.91.116.6:4975 , piece_id=D4MDIFGGVQ6GXCBD3EE4FC4XEEEI6XT2TSDJV4VX6G62TZA4SENA, seg_pos={0 0}, upload=459611]: Long tail cancellation due to optimal threshold Dec 27 11:25:08.660326952 (523h56m8.531409987s) [seg_pos={0 0}, upload=459611]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:25:08.660409599 (523h56m8.531492644s) [upload=459611]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:25:08.660416703 (523h56m8.531499738s) [upload=459611]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:25:08.950969645 (523h56m8.82205268s) [upload=459611]: Done uploading segments... Dec 27 11:25:08.950989223 (523h56m8.822072268s) [upload=459611]: Upload finished. err: <nil> Duration: 5.594776208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X] duration: 30.003168376s trace: https://jaeger.dev.storj.io/trace/42d146cbf8df32b5 FAIL (2025-12-27 11:25:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 11:25:06.167671957 (523h56m6.038755002s) [upload=459611]: Starting upload Dec 27 11:25:06.167757741 (523h56m6.038840776s) [upload=459611]: Done starting upload Dec 27 11:25:06.167863993 (523h56m6.038947048s) [upload=459611]: Uploading segments... Dec 27 11:25:06.168079964 (523h56m6.039163009s) [upload=459611]: Got next segment. Inline: false Dec 27 11:25:06.168145319 (523h56m6.039228364s) [upload=459611]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:25:06.168154135 (523h56m6.03923718s) [upload=459611]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:25:06.421674230 (523h56m6.292757275s) [seg_pos={0 0}, upload=459611]: Begin upload segment... Dec 27 11:25:06.421738753 (523h56m6.292821798s) [seg_pos={0 0}, upload=459611]: Done begin upload segment. Dec 27 11:25:06.421916251 (523h56m6.292999296s) [noise=true , address=145.79.227.50:4375 , piece_id=4NXNAYZZUHVPIAXI54U4WVUWZJQCE3VIDQ7SUXJ6CBQJ7ZQAHJFA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.421980444 (523h56m6.293063479s) [noise=true , address=145.79.227.46:4602 , piece_id=DYDB6H75QHHQNI5A75DUCFNC4JM5E5IYEV4GP34MZWXD6FWALOVA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.422671820 (523h56m6.293754865s) [noise=true , address=77.90.31.21:4878 , piece_id=LNS6AMLEKQEIF5JEN6UACQVFSGRG6HAS5W4MBOJAW6KDYQO5CM7A, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.422926996 (523h56m6.294010041s) [noise=true , address=45.91.116.11:4120 , piece_id=K5RLQYJBNOKM27GOBVGDIV7O4IRXMHMMEKBP7YYFE62NG6BO5CSQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.423461373 (523h56m6.294544418s) [noise=true , address=145.79.227.42:4123 , piece_id=53XNDYY5RLAQIF5HR4FBBLHXZTIJU65QN52FOAYXVCZT2RSCCMNA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.423864100 (523h56m6.294947145s) [noise=true , address=145.79.227.5:4221 , piece_id=NVQZU2Z7FOGABIX6ALMXCQIQBFYDSA2DEYDHK5TNPOPLXN7ARK6Q, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.424272348 (523h56m6.295355393s) [noise=true , address=77.90.31.14:4661 , piece_id=T77SHWDIOKJ2ULPQ577XDWZSQZ6RHTDSK7IVYUSIHZDZEMR3OGXQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.424914701 (523h56m6.295997746s) [noise=true , address=145.79.227.20:4438 , piece_id=DGGSSDMZN6GPWT7E45MXDLN2HIH5WOANOZO6PYP4EPGIWKH3XB4Q, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.425344961 (523h56m6.296428006s) [noise=true , address=145.79.227.2:4113 , piece_id=QK6SBPDK6QGMG5USMQBF4LLKU6IKTZ2J2LRONDQX4X2MY6I63IPA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.425847427 (523h56m6.296930482s) [noise=true , address=45.91.116.6:4975 , piece_id=D4MDIFGGVQ6GXCBD3EE4FC4XEEEI6XT2TSDJV4VX6G62TZA4SENA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.430273525 (523h56m6.30135656s) [noise=true , address=145.79.227.4:4186 , piece_id=GHDLMEG4UOKIHNBA3NZY7XBAMUZD5A3KD6QUUS2V4SWQDJE2K2LA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.430866284 (523h56m6.301949329s) [noise=true , address=77.90.31.10:4517 , piece_id=F7JVENE5H6CIRARN5LLG4VKLGMKB6ZRDBHKCJXSUOP6Z26FYZILA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.431369062 (523h56m6.302452107s) [noise=true , address=145.79.227.13:4802 , piece_id=Z75VG647RDW43724SKOQVE7J6EYMXI634R2SIS36DZSECVMMU7FQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.431871428 (523h56m6.302954473s) [noise=true , address=145.79.227.14:4837 , piece_id=4AHK2EPGMNXRCFR3TUEMAM2ZXE3ZAI6DYDGLTXB553QI46FOTJWA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.432809594 (523h56m6.303892629s) [noise=true , address=145.79.227.38:4978 , piece_id=UDAGYOMY3WVVS6SUXKIUPXUEHC5E2KMV3TDYTXJT33E25WTQBIZA, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:06.435090277 (523h56m6.306173323s) [noise=true , address=145.79.227.41:4085 , piece_id=UYLG3PUNGDR6IKEKB43YWITHSCDA5JV2ND5FLC26MKHBRBYMAFVQ, seg_pos={0 0}, upload=459611]: Uploading piece... Dec 27 11:25:08.047099283 (523h56m7.918182328s) [upload=459611]: Next returned nil segment Dec 27 11:25:08.047112739 (523h56m7.918195774s) [upload=459611]: Waiting for error group managing segments... Dec 27 11:25:08.430945751 (523h56m8.302028796s) [noise=true , address=145.79.227.14:4837 , piece_id=4AHK2EPGMNXRCFR3TUEMAM2ZXE3ZAI6DYDGLTXB553QI46FOTJWA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.431725796 (523h56m8.302808831s) [noise=true , address=145.79.227.38:4978 , piece_id=UDAGYOMY3WVVS6SUXKIUPXUEHC5E2KMV3TDYTXJT33E25WTQBIZA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.435841935 (523h56m8.30692497s) [noise=true , address=145.79.227.20:4438 , piece_id=DGGSSDMZN6GPWT7E45MXDLN2HIH5WOANOZO6PYP4EPGIWKH3XB4Q, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.436153698 (523h56m8.307236733s) [noise=true , address=145.79.227.2:4113 , piece_id=QK6SBPDK6QGMG5USMQBF4LLKU6IKTZ2J2LRONDQX4X2MY6I63IPA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.443356844 (523h56m8.314439879s) [noise=true , address=145.79.227.50:4375 , piece_id=4NXNAYZZUHVPIAXI54U4WVUWZJQCE3VIDQ7SUXJ6CBQJ7ZQAHJFA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.446780423 (523h56m8.317863459s) [noise=true , address=145.79.227.41:4085 , piece_id=UYLG3PUNGDR6IKEKB43YWITHSCDA5JV2ND5FLC26MKHBRBYMAFVQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.449574013 (523h56m8.320657058s) [noise=true , address=145.79.227.5:4221 , piece_id=NVQZU2Z7FOGABIX6ALMXCQIQBFYDSA2DEYDHK5TNPOPLXN7ARK6Q, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.453222270 (523h56m8.324305315s) [noise=true , address=145.79.227.46:4602 , piece_id=DYDB6H75QHHQNI5A75DUCFNC4JM5E5IYEV4GP34MZWXD6FWALOVA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.459121232 (523h56m8.330204267s) [noise=true , address=145.79.227.4:4186 , piece_id=GHDLMEG4UOKIHNBA3NZY7XBAMUZD5A3KD6QUUS2V4SWQDJE2K2LA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.460885442 (523h56m8.331968477s) [noise=true , address=145.79.227.42:4123 , piece_id=53XNDYY5RLAQIF5HR4FBBLHXZTIJU65QN52FOAYXVCZT2RSCCMNA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.466587089 (523h56m8.337670134s) [noise=true , address=145.79.227.13:4802 , piece_id=Z75VG647RDW43724SKOQVE7J6EYMXI634R2SIS36DZSECVMMU7FQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.611439863 (523h56m8.482522898s) [noise=true , address=77.90.31.14:4661 , piece_id=T77SHWDIOKJ2ULPQ577XDWZSQZ6RHTDSK7IVYUSIHZDZEMR3OGXQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.659568017 (523h56m8.530651062s) [noise=true , address=77.90.31.21:4878 , piece_id=LNS6AMLEKQEIF5JEN6UACQVFSGRG6HAS5W4MBOJAW6KDYQO5CM7A, seg_pos={0 0}, upload=459611]: Done uploading piece. err: <nil> Dec 27 11:25:08.659592274 (523h56m8.530675309s) [seg_pos={0 0}, upload=459611]: Segment reached optimal threshold of 13 pieces. Dec 27 11:25:08.659804528 (523h56m8.530887573s) [noise=true , address=77.90.31.10:4517 , piece_id=F7JVENE5H6CIRARN5LLG4VKLGMKB6ZRDBHKCJXSUOP6Z26FYZILA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 11:25:08.659814256 (523h56m8.530897301s) [noise=true , address=77.90.31.10:4517 , piece_id=F7JVENE5H6CIRARN5LLG4VKLGMKB6ZRDBHKCJXSUOP6Z26FYZILA, seg_pos={0 0}, upload=459611]: Long tail cancellation due to optimal threshold Dec 27 11:25:08.660005109 (523h56m8.531088154s) [noise=true , address=45.91.116.11:4120 , piece_id=K5RLQYJBNOKM27GOBVGDIV7O4IRXMHMMEKBP7YYFE62NG6BO5CSQ, seg_pos={0 0}, upload=459611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 11:25:08.660013415 (523h56m8.53109645s) [noise=true , address=45.91.116.11:4120 , piece_id=K5RLQYJBNOKM27GOBVGDIV7O4IRXMHMMEKBP7YYFE62NG6BO5CSQ, seg_pos={0 0}, upload=459611]: Long tail cancellation due to optimal threshold Dec 27 11:25:08.660296905 (523h56m8.53137994s) [noise=true , address=45.91.116.6:4975 , piece_id=D4MDIFGGVQ6GXCBD3EE4FC4XEEEI6XT2TSDJV4VX6G62TZA4SENA, seg_pos={0 0}, upload=459611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:32912->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 11:25:08.660305922 (523h56m8.531388967s) [noise=true , address=45.91.116.6:4975 , piece_id=D4MDIFGGVQ6GXCBD3EE4FC4XEEEI6XT2TSDJV4VX6G62TZA4SENA, seg_pos={0 0}, upload=459611]: Long tail cancellation due to optimal threshold Dec 27 11:25:08.660326952 (523h56m8.531409987s) [seg_pos={0 0}, upload=459611]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:25:08.660409599 (523h56m8.531492644s) [upload=459611]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:25:08.660416703 (523h56m8.531499738s) [upload=459611]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:25:08.950969645 (523h56m8.82205268s) [upload=459611]: Done uploading segments... Dec 27 11:25:08.950989223 (523h56m8.822072268s) [upload=459611]: Upload finished. err: <nil> Duration: 30.003194134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..] duration: 5.916285123s trace: https://jaeger.dev.storj.io/trace/286dd7404bd53819 FAIL (2025-12-27 11:24:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ) context deadline exceeded ======LOGS====== Dec 27 11:21:58.607602738 (523h52m58.478685773s) [upload=459568]: Starting upload Dec 27 11:21:58.607680746 (523h52m58.478763781s) [upload=459568]: Done starting upload Dec 27 11:21:58.607718859 (523h52m58.478801894s) [upload=459568]: Uploading segments... Dec 27 11:21:58.607910303 (523h52m58.478993348s) [upload=459568]: Got next segment. Inline: false Dec 27 11:21:58.607963796 (523h52m58.479046841s) [upload=459568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:21:58.607971330 (523h52m58.479054375s) [upload=459568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:21:58.861416451 (523h52m58.732499497s) [seg_pos={0 0}, upload=459568]: Begin upload segment... Dec 27 11:21:58.861467058 (523h52m58.732550093s) [seg_pos={0 0}, upload=459568]: Done begin upload segment. Dec 27 11:21:58.861496104 (523h52m58.732579149s) [noise=true , address=145.79.227.50:4373 , piece_id=WHN6GFBMSHGGGGPUPD7OBHM3KYVOLJVL6RGCWDFWKG2SVPY2OSSA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.865671794 (523h52m58.736754859s) [noise=true , address=145.79.227.38:4978 , piece_id=EK6ZKSRTK3WGU2T7JQTPPQRFSNNYJCMZ7LL4CCE5VSRK6GITNS6A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.867053175 (523h52m58.73813621s) [noise=true , address=45.91.116.10:4082 , piece_id=5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.868450315 (523h52m58.73953335s) [noise=true , address=145.79.227.4:4185 , piece_id=KLJFJX32QZJP6PPG2PJGIDAWJXJDPTXV5BVDJ4NJWZEECXVQ5ODA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.868944647 (523h52m58.740027682s) [noise=true , address=145.79.227.3:4149 , piece_id=SITHSC6JO25LIZTJZEA4BPTICYIJ4K7QCTFYDZ323NYVBGZRILGA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870325125 (523h52m58.74140817s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870605870 (523h52m58.741688915s) [noise=true , address=145.79.227.7:4295 , piece_id=LTFKYG7V4EG4BQ3HRRC6JSQJTEYMP2ZLQZGCFBYMIYVO6V7GOIQA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870974592 (523h52m58.742057637s) [noise=true , address=145.79.227.20:4437 , piece_id=RL3MUFG4V5XD76O2JAYP7R6NHCKUZ57QQGIGYYM5HSD4NSKS7Y2Q, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.872243077 (523h52m58.743326112s) [noise=true , address=145.79.227.11:4729 , piece_id=XVM73RYIS75A25UDKUESGZDM4XYGI5V7WNWLGRQSROJCSFUQ7KQQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.873158922 (523h52m58.744241957s) [noise=true , address=145.79.227.47:4266 , piece_id=LZA7LAB4YR2NZY3VTPHCZJGZOCPB664XBI4NNLQMWHWL5AT2YLBA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.873703939 (523h52m58.744786974s) [noise=true , address=145.79.227.16:4912 , piece_id=EF5KQWAQOKZGBUG7ZCY3NE3ICUHEDDDA4COG4H7NC2G7PLYES6NA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.874433989 (523h52m58.745517034s) [noise=true , address=77.90.31.10:4520 , piece_id=ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.874999726 (523h52m58.746082771s) [noise=true , address=145.79.227.52:4447 , piece_id=KN5RHF6MU4FBQWGYOCEVMEP2626JVLGKSOM35RTHNCGM2GWS753A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.876272681 (523h52m58.747355716s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.877356194 (523h52m58.748439239s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.878088789 (523h52m58.749171844s) [noise=true , address=45.91.116.9:4264 , piece_id=X243SN7K7HFBZCDHKFOSQSAAQFTNOLL775RFCFFFRAGGZNJHG7LA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:22:00.584233065 (523h53m0.4553161s) [upload=459568]: Next returned nil segment Dec 27 11:22:00.584273361 (523h53m0.455356396s) [upload=459568]: Waiting for error group managing segments... Dec 27 11:22:00.870279025 (523h53m0.74136206s) [noise=true , address=145.79.227.20:4437 , piece_id=RL3MUFG4V5XD76O2JAYP7R6NHCKUZ57QQGIGYYM5HSD4NSKS7Y2Q, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.874708679 (523h53m0.745791724s) [noise=true , address=145.79.227.7:4295 , piece_id=LTFKYG7V4EG4BQ3HRRC6JSQJTEYMP2ZLQZGCFBYMIYVO6V7GOIQA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.876312073 (523h53m0.747395128s) [noise=true , address=145.79.227.50:4373 , piece_id=WHN6GFBMSHGGGGPUPD7OBHM3KYVOLJVL6RGCWDFWKG2SVPY2OSSA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.880601481 (523h53m0.751684526s) [noise=true , address=145.79.227.52:4447 , piece_id=KN5RHF6MU4FBQWGYOCEVMEP2626JVLGKSOM35RTHNCGM2GWS753A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.884708982 (523h53m0.755792027s) [noise=true , address=145.79.227.16:4912 , piece_id=EF5KQWAQOKZGBUG7ZCY3NE3ICUHEDDDA4COG4H7NC2G7PLYES6NA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.890190109 (523h53m0.761273154s) [noise=true , address=145.79.227.38:4978 , piece_id=EK6ZKSRTK3WGU2T7JQTPPQRFSNNYJCMZ7LL4CCE5VSRK6GITNS6A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.893052389 (523h53m0.764135434s) [noise=true , address=145.79.227.47:4266 , piece_id=LZA7LAB4YR2NZY3VTPHCZJGZOCPB664XBI4NNLQMWHWL5AT2YLBA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.902469481 (523h53m0.773552526s) [noise=true , address=145.79.227.4:4185 , piece_id=KLJFJX32QZJP6PPG2PJGIDAWJXJDPTXV5BVDJ4NJWZEECXVQ5ODA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.907133262 (523h53m0.778216297s) [noise=true , address=145.79.227.3:4149 , piece_id=SITHSC6JO25LIZTJZEA4BPTICYIJ4K7QCTFYDZ323NYVBGZRILGA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.916782235 (523h53m0.78786527s) [noise=true , address=145.79.227.11:4729 , piece_id=XVM73RYIS75A25UDKUESGZDM4XYGI5V7WNWLGRQSROJCSFUQ7KQQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.056999131 (523h53m0.928082176s) [noise=true , address=77.90.31.10:4520 , piece_id=ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.094826882 (523h53m0.965909917s) [noise=true , address=45.91.116.9:4264 , piece_id=X243SN7K7HFBZCDHKFOSQSAAQFTNOLL775RFCFFFRAGGZNJHG7LA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.099399128 (523h53m0.970482173s) [noise=true , address=45.91.116.10:4082 , piece_id=5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.099426240 (523h53m0.970509285s) [seg_pos={0 0}, upload=459568]: Segment reached optimal threshold of 13 pieces. Dec 27 11:22:01.099707515 (523h53m0.97079055s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection Dec 27 11:22:01.099738655 (523h53m0.9708217s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.099808588 (523h53m0.970891623s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 11:22:01.099827995 (523h53m0.97091104s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.100006765 (523h53m0.971089801s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 11:22:01.100014490 (523h53m0.971097535s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.100040139 (523h53m0.971123184s) [seg_pos={0 0}, upload=459568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:22:01.100119710 (523h53m0.971202745s) [upload=459568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:22:01.100125231 (523h53m0.971208266s) [upload=459568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:22:01.387490713 (523h53m1.258573768s) [upload=459568]: Done uploading segments... Dec 27 11:22:01.387519288 (523h53m1.258602333s) [upload=459568]: Upload finished. err: <nil> Duration: 5.916313678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X.] duration: 5.626226543s trace: https://jaeger.dev.storj.io/trace/286dd7404bd53819 FAIL (2025-12-27 11:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ) context deadline exceeded ======LOGS====== Dec 27 11:21:58.607602738 (523h52m58.478685773s) [upload=459568]: Starting upload Dec 27 11:21:58.607680746 (523h52m58.478763781s) [upload=459568]: Done starting upload Dec 27 11:21:58.607718859 (523h52m58.478801894s) [upload=459568]: Uploading segments... Dec 27 11:21:58.607910303 (523h52m58.478993348s) [upload=459568]: Got next segment. Inline: false Dec 27 11:21:58.607963796 (523h52m58.479046841s) [upload=459568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:21:58.607971330 (523h52m58.479054375s) [upload=459568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:21:58.861416451 (523h52m58.732499497s) [seg_pos={0 0}, upload=459568]: Begin upload segment... Dec 27 11:21:58.861467058 (523h52m58.732550093s) [seg_pos={0 0}, upload=459568]: Done begin upload segment. Dec 27 11:21:58.861496104 (523h52m58.732579149s) [noise=true , address=145.79.227.50:4373 , piece_id=WHN6GFBMSHGGGGPUPD7OBHM3KYVOLJVL6RGCWDFWKG2SVPY2OSSA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.865671794 (523h52m58.736754859s) [noise=true , address=145.79.227.38:4978 , piece_id=EK6ZKSRTK3WGU2T7JQTPPQRFSNNYJCMZ7LL4CCE5VSRK6GITNS6A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.867053175 (523h52m58.73813621s) [noise=true , address=45.91.116.10:4082 , piece_id=5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.868450315 (523h52m58.73953335s) [noise=true , address=145.79.227.4:4185 , piece_id=KLJFJX32QZJP6PPG2PJGIDAWJXJDPTXV5BVDJ4NJWZEECXVQ5ODA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.868944647 (523h52m58.740027682s) [noise=true , address=145.79.227.3:4149 , piece_id=SITHSC6JO25LIZTJZEA4BPTICYIJ4K7QCTFYDZ323NYVBGZRILGA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870325125 (523h52m58.74140817s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870605870 (523h52m58.741688915s) [noise=true , address=145.79.227.7:4295 , piece_id=LTFKYG7V4EG4BQ3HRRC6JSQJTEYMP2ZLQZGCFBYMIYVO6V7GOIQA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870974592 (523h52m58.742057637s) [noise=true , address=145.79.227.20:4437 , piece_id=RL3MUFG4V5XD76O2JAYP7R6NHCKUZ57QQGIGYYM5HSD4NSKS7Y2Q, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.872243077 (523h52m58.743326112s) [noise=true , address=145.79.227.11:4729 , piece_id=XVM73RYIS75A25UDKUESGZDM4XYGI5V7WNWLGRQSROJCSFUQ7KQQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.873158922 (523h52m58.744241957s) [noise=true , address=145.79.227.47:4266 , piece_id=LZA7LAB4YR2NZY3VTPHCZJGZOCPB664XBI4NNLQMWHWL5AT2YLBA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.873703939 (523h52m58.744786974s) [noise=true , address=145.79.227.16:4912 , piece_id=EF5KQWAQOKZGBUG7ZCY3NE3ICUHEDDDA4COG4H7NC2G7PLYES6NA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.874433989 (523h52m58.745517034s) [noise=true , address=77.90.31.10:4520 , piece_id=ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.874999726 (523h52m58.746082771s) [noise=true , address=145.79.227.52:4447 , piece_id=KN5RHF6MU4FBQWGYOCEVMEP2626JVLGKSOM35RTHNCGM2GWS753A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.876272681 (523h52m58.747355716s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.877356194 (523h52m58.748439239s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.878088789 (523h52m58.749171844s) [noise=true , address=45.91.116.9:4264 , piece_id=X243SN7K7HFBZCDHKFOSQSAAQFTNOLL775RFCFFFRAGGZNJHG7LA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:22:00.584233065 (523h53m0.4553161s) [upload=459568]: Next returned nil segment Dec 27 11:22:00.584273361 (523h53m0.455356396s) [upload=459568]: Waiting for error group managing segments... Dec 27 11:22:00.870279025 (523h53m0.74136206s) [noise=true , address=145.79.227.20:4437 , piece_id=RL3MUFG4V5XD76O2JAYP7R6NHCKUZ57QQGIGYYM5HSD4NSKS7Y2Q, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.874708679 (523h53m0.745791724s) [noise=true , address=145.79.227.7:4295 , piece_id=LTFKYG7V4EG4BQ3HRRC6JSQJTEYMP2ZLQZGCFBYMIYVO6V7GOIQA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.876312073 (523h53m0.747395128s) [noise=true , address=145.79.227.50:4373 , piece_id=WHN6GFBMSHGGGGPUPD7OBHM3KYVOLJVL6RGCWDFWKG2SVPY2OSSA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.880601481 (523h53m0.751684526s) [noise=true , address=145.79.227.52:4447 , piece_id=KN5RHF6MU4FBQWGYOCEVMEP2626JVLGKSOM35RTHNCGM2GWS753A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.884708982 (523h53m0.755792027s) [noise=true , address=145.79.227.16:4912 , piece_id=EF5KQWAQOKZGBUG7ZCY3NE3ICUHEDDDA4COG4H7NC2G7PLYES6NA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.890190109 (523h53m0.761273154s) [noise=true , address=145.79.227.38:4978 , piece_id=EK6ZKSRTK3WGU2T7JQTPPQRFSNNYJCMZ7LL4CCE5VSRK6GITNS6A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.893052389 (523h53m0.764135434s) [noise=true , address=145.79.227.47:4266 , piece_id=LZA7LAB4YR2NZY3VTPHCZJGZOCPB664XBI4NNLQMWHWL5AT2YLBA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.902469481 (523h53m0.773552526s) [noise=true , address=145.79.227.4:4185 , piece_id=KLJFJX32QZJP6PPG2PJGIDAWJXJDPTXV5BVDJ4NJWZEECXVQ5ODA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.907133262 (523h53m0.778216297s) [noise=true , address=145.79.227.3:4149 , piece_id=SITHSC6JO25LIZTJZEA4BPTICYIJ4K7QCTFYDZ323NYVBGZRILGA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.916782235 (523h53m0.78786527s) [noise=true , address=145.79.227.11:4729 , piece_id=XVM73RYIS75A25UDKUESGZDM4XYGI5V7WNWLGRQSROJCSFUQ7KQQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.056999131 (523h53m0.928082176s) [noise=true , address=77.90.31.10:4520 , piece_id=ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.094826882 (523h53m0.965909917s) [noise=true , address=45.91.116.9:4264 , piece_id=X243SN7K7HFBZCDHKFOSQSAAQFTNOLL775RFCFFFRAGGZNJHG7LA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.099399128 (523h53m0.970482173s) [noise=true , address=45.91.116.10:4082 , piece_id=5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.099426240 (523h53m0.970509285s) [seg_pos={0 0}, upload=459568]: Segment reached optimal threshold of 13 pieces. Dec 27 11:22:01.099707515 (523h53m0.97079055s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection Dec 27 11:22:01.099738655 (523h53m0.9708217s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.099808588 (523h53m0.970891623s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 11:22:01.099827995 (523h53m0.97091104s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.100006765 (523h53m0.971089801s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 11:22:01.100014490 (523h53m0.971097535s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.100040139 (523h53m0.971123184s) [seg_pos={0 0}, upload=459568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:22:01.100119710 (523h53m0.971202745s) [upload=459568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:22:01.100125231 (523h53m0.971208266s) [upload=459568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:22:01.387490713 (523h53m1.258573768s) [upload=459568]: Done uploading segments... Dec 27 11:22:01.387519288 (523h53m1.258602333s) [upload=459568]: Upload finished. err: <nil> Duration: 5.626258313s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X] duration: 30.00586775s trace: https://jaeger.dev.storj.io/trace/286dd7404bd53819 FAIL (2025-12-27 11:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ) context deadline exceeded ======LOGS====== Dec 27 11:21:58.607602738 (523h52m58.478685773s) [upload=459568]: Starting upload Dec 27 11:21:58.607680746 (523h52m58.478763781s) [upload=459568]: Done starting upload Dec 27 11:21:58.607718859 (523h52m58.478801894s) [upload=459568]: Uploading segments... Dec 27 11:21:58.607910303 (523h52m58.478993348s) [upload=459568]: Got next segment. Inline: false Dec 27 11:21:58.607963796 (523h52m58.479046841s) [upload=459568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:21:58.607971330 (523h52m58.479054375s) [upload=459568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:21:58.861416451 (523h52m58.732499497s) [seg_pos={0 0}, upload=459568]: Begin upload segment... Dec 27 11:21:58.861467058 (523h52m58.732550093s) [seg_pos={0 0}, upload=459568]: Done begin upload segment. Dec 27 11:21:58.861496104 (523h52m58.732579149s) [noise=true , address=145.79.227.50:4373 , piece_id=WHN6GFBMSHGGGGPUPD7OBHM3KYVOLJVL6RGCWDFWKG2SVPY2OSSA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.865671794 (523h52m58.736754859s) [noise=true , address=145.79.227.38:4978 , piece_id=EK6ZKSRTK3WGU2T7JQTPPQRFSNNYJCMZ7LL4CCE5VSRK6GITNS6A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.867053175 (523h52m58.73813621s) [noise=true , address=45.91.116.10:4082 , piece_id=5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.868450315 (523h52m58.73953335s) [noise=true , address=145.79.227.4:4185 , piece_id=KLJFJX32QZJP6PPG2PJGIDAWJXJDPTXV5BVDJ4NJWZEECXVQ5ODA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.868944647 (523h52m58.740027682s) [noise=true , address=145.79.227.3:4149 , piece_id=SITHSC6JO25LIZTJZEA4BPTICYIJ4K7QCTFYDZ323NYVBGZRILGA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870325125 (523h52m58.74140817s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870605870 (523h52m58.741688915s) [noise=true , address=145.79.227.7:4295 , piece_id=LTFKYG7V4EG4BQ3HRRC6JSQJTEYMP2ZLQZGCFBYMIYVO6V7GOIQA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.870974592 (523h52m58.742057637s) [noise=true , address=145.79.227.20:4437 , piece_id=RL3MUFG4V5XD76O2JAYP7R6NHCKUZ57QQGIGYYM5HSD4NSKS7Y2Q, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.872243077 (523h52m58.743326112s) [noise=true , address=145.79.227.11:4729 , piece_id=XVM73RYIS75A25UDKUESGZDM4XYGI5V7WNWLGRQSROJCSFUQ7KQQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.873158922 (523h52m58.744241957s) [noise=true , address=145.79.227.47:4266 , piece_id=LZA7LAB4YR2NZY3VTPHCZJGZOCPB664XBI4NNLQMWHWL5AT2YLBA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.873703939 (523h52m58.744786974s) [noise=true , address=145.79.227.16:4912 , piece_id=EF5KQWAQOKZGBUG7ZCY3NE3ICUHEDDDA4COG4H7NC2G7PLYES6NA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.874433989 (523h52m58.745517034s) [noise=true , address=77.90.31.10:4520 , piece_id=ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.874999726 (523h52m58.746082771s) [noise=true , address=145.79.227.52:4447 , piece_id=KN5RHF6MU4FBQWGYOCEVMEP2626JVLGKSOM35RTHNCGM2GWS753A, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.876272681 (523h52m58.747355716s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.877356194 (523h52m58.748439239s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:21:58.878088789 (523h52m58.749171844s) [noise=true , address=45.91.116.9:4264 , piece_id=X243SN7K7HFBZCDHKFOSQSAAQFTNOLL775RFCFFFRAGGZNJHG7LA, seg_pos={0 0}, upload=459568]: Uploading piece... Dec 27 11:22:00.584233065 (523h53m0.4553161s) [upload=459568]: Next returned nil segment Dec 27 11:22:00.584273361 (523h53m0.455356396s) [upload=459568]: Waiting for error group managing segments... Dec 27 11:22:00.870279025 (523h53m0.74136206s) [noise=true , address=145.79.227.20:4437 , piece_id=RL3MUFG4V5XD76O2JAYP7R6NHCKUZ57QQGIGYYM5HSD4NSKS7Y2Q, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.874708679 (523h53m0.745791724s) [noise=true , address=145.79.227.7:4295 , piece_id=LTFKYG7V4EG4BQ3HRRC6JSQJTEYMP2ZLQZGCFBYMIYVO6V7GOIQA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.876312073 (523h53m0.747395128s) [noise=true , address=145.79.227.50:4373 , piece_id=WHN6GFBMSHGGGGPUPD7OBHM3KYVOLJVL6RGCWDFWKG2SVPY2OSSA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.880601481 (523h53m0.751684526s) [noise=true , address=145.79.227.52:4447 , piece_id=KN5RHF6MU4FBQWGYOCEVMEP2626JVLGKSOM35RTHNCGM2GWS753A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.884708982 (523h53m0.755792027s) [noise=true , address=145.79.227.16:4912 , piece_id=EF5KQWAQOKZGBUG7ZCY3NE3ICUHEDDDA4COG4H7NC2G7PLYES6NA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.890190109 (523h53m0.761273154s) [noise=true , address=145.79.227.38:4978 , piece_id=EK6ZKSRTK3WGU2T7JQTPPQRFSNNYJCMZ7LL4CCE5VSRK6GITNS6A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.893052389 (523h53m0.764135434s) [noise=true , address=145.79.227.47:4266 , piece_id=LZA7LAB4YR2NZY3VTPHCZJGZOCPB664XBI4NNLQMWHWL5AT2YLBA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.902469481 (523h53m0.773552526s) [noise=true , address=145.79.227.4:4185 , piece_id=KLJFJX32QZJP6PPG2PJGIDAWJXJDPTXV5BVDJ4NJWZEECXVQ5ODA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.907133262 (523h53m0.778216297s) [noise=true , address=145.79.227.3:4149 , piece_id=SITHSC6JO25LIZTJZEA4BPTICYIJ4K7QCTFYDZ323NYVBGZRILGA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:00.916782235 (523h53m0.78786527s) [noise=true , address=145.79.227.11:4729 , piece_id=XVM73RYIS75A25UDKUESGZDM4XYGI5V7WNWLGRQSROJCSFUQ7KQQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.056999131 (523h53m0.928082176s) [noise=true , address=77.90.31.10:4520 , piece_id=ZUID7YAKHGUPDEDQHL67WWXDZ4TRJE3NDKHESORNETB4RH3MJ52A, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.094826882 (523h53m0.965909917s) [noise=true , address=45.91.116.9:4264 , piece_id=X243SN7K7HFBZCDHKFOSQSAAQFTNOLL775RFCFFFRAGGZNJHG7LA, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.099399128 (523h53m0.970482173s) [noise=true , address=45.91.116.10:4082 , piece_id=5AVJB6EXEUALWZLYEMKNLTSST32IVEXR7U7LUHFLV3LZ2XP56KFQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: <nil> Dec 27 11:22:01.099426240 (523h53m0.970509285s) [seg_pos={0 0}, upload=459568]: Segment reached optimal threshold of 13 pieces. Dec 27 11:22:01.099707515 (523h53m0.97079055s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56128->77.90.31.17:4770: use of closed network connection Dec 27 11:22:01.099738655 (523h53m0.9708217s) [noise=true , address=77.90.31.17:4770 , piece_id=SF3TQMRCBAN2VOCA3R7BSQ65VKRBDMNJXAIGTUBVGHDYQTOREWIQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.099808588 (523h53m0.970891623s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 11:22:01.099827995 (523h53m0.97091104s) [noise=true , address=145.79.227.40:4051 , piece_id=NELSQX7TWX4LNHFJJJ5KYW7HDOTGLXRQBFNFLQPN7HOXY4GJLDYQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.100006765 (523h53m0.971089801s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 11:22:01.100014490 (523h53m0.971097535s) [noise=true , address=45.91.116.8:4048 , piece_id=QEAQX6JC4AR6GX3HO6DQD3TJIC7MPL3RVBUH47EI3SO4EUNOAQUQ, seg_pos={0 0}, upload=459568]: Long tail cancellation due to optimal threshold Dec 27 11:22:01.100040139 (523h53m0.971123184s) [seg_pos={0 0}, upload=459568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:22:01.100119710 (523h53m0.971202745s) [upload=459568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:22:01.100125231 (523h53m0.971208266s) [upload=459568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:22:01.387490713 (523h53m1.258573768s) [upload=459568]: Done uploading segments... Dec 27 11:22:01.387519288 (523h53m1.258602333s) [upload=459568]: Upload finished. err: <nil> Duration: 30.00596825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...] duration: 6.142911888s trace: https://jaeger.dev.storj.io/trace/51cbf38c3d4e9dfb FAIL (2025-12-27 11:21:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ) context deadline exceeded ======LOGS====== Dec 27 11:18:41.856771094 (523h49m41.727854129s) [upload=459525]: Starting upload Dec 27 11:18:41.856840466 (523h49m41.727923531s) [upload=459525]: Done starting upload Dec 27 11:18:41.856955636 (523h49m41.728038681s) [upload=459525]: Uploading segments... Dec 27 11:18:41.857101133 (523h49m41.728184208s) [upload=459525]: Got next segment. Inline: false Dec 27 11:18:41.857135749 (523h49m41.728218784s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:18:41.857141780 (523h49m41.728224825s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:18:42.102279323 (523h49m41.973362358s) [seg_pos={0 0}, upload=459525]: Begin upload segment... Dec 27 11:18:42.102319920 (523h49m41.973402955s) [seg_pos={0 0}, upload=459525]: Done begin upload segment. Dec 27 11:18:42.102355638 (523h49m41.973438673s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102363613 (523h49m41.973446648s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102788452 (523h49m41.973871487s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103214694 (523h49m41.97429773s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103648410 (523h49m41.974731445s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103984731 (523h49m41.975067766s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104248602 (523h49m41.975331648s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104566207 (523h49m41.975649252s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106012291 (523h49m41.977095356s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106275342 (523h49m41.977358377s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106393948 (523h49m41.977476983s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106750467 (523h49m41.977833502s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107103769 (523h49m41.978186804s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107409381 (523h49m41.978492416s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107857575 (523h49m41.97894061s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.109167149 (523h49m41.980250184s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:43.813337727 (523h49m43.684420772s) [upload=459525]: Next returned nil segment Dec 27 11:18:43.813356433 (523h49m43.684439468s) [upload=459525]: Waiting for error group managing segments... Dec 27 11:18:44.097152137 (523h49m43.968235172s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.110645221 (523h49m43.981728256s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.111175320 (523h49m43.982258355s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.112899472 (523h49m43.983982518s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.114278359 (523h49m43.985361394s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.118051503 (523h49m43.989134538s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.121413054 (523h49m43.992496089s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.133847121 (523h49m44.004930156s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.304328606 (523h49m44.175411642s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.346564811 (523h49m44.217647846s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.489411776 (523h49m44.360494821s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.667495207 (523h49m44.538578242s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852043327 (523h49m44.723126372s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852061782 (523h49m44.723144817s) [seg_pos={0 0}, upload=459525]: Segment reached optimal threshold of 13 pieces. Dec 27 11:18:44.852259720 (523h49m44.723342755s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852274067 (523h49m44.723357102s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852499085 (523h49m44.72358212s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852505407 (523h49m44.723588452s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852601680 (523h49m44.723684745s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852622801 (523h49m44.723705846s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852654441 (523h49m44.723737496s) [seg_pos={0 0}, upload=459525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:18:44.852810758 (523h49m44.723893803s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:18:44.852818463 (523h49m44.723901509s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:18:45.138122149 (523h49m45.009205184s) [upload=459525]: Done uploading segments... Dec 27 11:18:45.138140855 (523h49m45.00922389s) [upload=459525]: Upload finished. err: <nil> Duration: 6.14294444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X..] duration: 5.884887026s trace: https://jaeger.dev.storj.io/trace/51cbf38c3d4e9dfb FAIL (2025-12-27 11:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ) context deadline exceeded ======LOGS====== Dec 27 11:18:41.856771094 (523h49m41.727854129s) [upload=459525]: Starting upload Dec 27 11:18:41.856840466 (523h49m41.727923531s) [upload=459525]: Done starting upload Dec 27 11:18:41.856955636 (523h49m41.728038681s) [upload=459525]: Uploading segments... Dec 27 11:18:41.857101133 (523h49m41.728184208s) [upload=459525]: Got next segment. Inline: false Dec 27 11:18:41.857135749 (523h49m41.728218784s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:18:41.857141780 (523h49m41.728224825s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:18:42.102279323 (523h49m41.973362358s) [seg_pos={0 0}, upload=459525]: Begin upload segment... Dec 27 11:18:42.102319920 (523h49m41.973402955s) [seg_pos={0 0}, upload=459525]: Done begin upload segment. Dec 27 11:18:42.102355638 (523h49m41.973438673s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102363613 (523h49m41.973446648s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102788452 (523h49m41.973871487s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103214694 (523h49m41.97429773s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103648410 (523h49m41.974731445s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103984731 (523h49m41.975067766s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104248602 (523h49m41.975331648s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104566207 (523h49m41.975649252s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106012291 (523h49m41.977095356s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106275342 (523h49m41.977358377s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106393948 (523h49m41.977476983s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106750467 (523h49m41.977833502s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107103769 (523h49m41.978186804s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107409381 (523h49m41.978492416s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107857575 (523h49m41.97894061s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.109167149 (523h49m41.980250184s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:43.813337727 (523h49m43.684420772s) [upload=459525]: Next returned nil segment Dec 27 11:18:43.813356433 (523h49m43.684439468s) [upload=459525]: Waiting for error group managing segments... Dec 27 11:18:44.097152137 (523h49m43.968235172s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.110645221 (523h49m43.981728256s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.111175320 (523h49m43.982258355s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.112899472 (523h49m43.983982518s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.114278359 (523h49m43.985361394s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.118051503 (523h49m43.989134538s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.121413054 (523h49m43.992496089s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.133847121 (523h49m44.004930156s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.304328606 (523h49m44.175411642s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.346564811 (523h49m44.217647846s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.489411776 (523h49m44.360494821s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.667495207 (523h49m44.538578242s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852043327 (523h49m44.723126372s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852061782 (523h49m44.723144817s) [seg_pos={0 0}, upload=459525]: Segment reached optimal threshold of 13 pieces. Dec 27 11:18:44.852259720 (523h49m44.723342755s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852274067 (523h49m44.723357102s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852499085 (523h49m44.72358212s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852505407 (523h49m44.723588452s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852601680 (523h49m44.723684745s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852622801 (523h49m44.723705846s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852654441 (523h49m44.723737496s) [seg_pos={0 0}, upload=459525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:18:44.852810758 (523h49m44.723893803s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:18:44.852818463 (523h49m44.723901509s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:18:45.138122149 (523h49m45.009205184s) [upload=459525]: Done uploading segments... Dec 27 11:18:45.138140855 (523h49m45.00922389s) [upload=459525]: Upload finished. err: <nil> Duration: 5.884918977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X.] duration: 5.650828728s trace: https://jaeger.dev.storj.io/trace/51cbf38c3d4e9dfb FAIL (2025-12-27 11:19:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ) context deadline exceeded ======LOGS====== Dec 27 11:18:41.856771094 (523h49m41.727854129s) [upload=459525]: Starting upload Dec 27 11:18:41.856840466 (523h49m41.727923531s) [upload=459525]: Done starting upload Dec 27 11:18:41.856955636 (523h49m41.728038681s) [upload=459525]: Uploading segments... Dec 27 11:18:41.857101133 (523h49m41.728184208s) [upload=459525]: Got next segment. Inline: false Dec 27 11:18:41.857135749 (523h49m41.728218784s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:18:41.857141780 (523h49m41.728224825s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:18:42.102279323 (523h49m41.973362358s) [seg_pos={0 0}, upload=459525]: Begin upload segment... Dec 27 11:18:42.102319920 (523h49m41.973402955s) [seg_pos={0 0}, upload=459525]: Done begin upload segment. Dec 27 11:18:42.102355638 (523h49m41.973438673s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102363613 (523h49m41.973446648s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102788452 (523h49m41.973871487s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103214694 (523h49m41.97429773s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103648410 (523h49m41.974731445s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103984731 (523h49m41.975067766s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104248602 (523h49m41.975331648s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104566207 (523h49m41.975649252s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106012291 (523h49m41.977095356s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106275342 (523h49m41.977358377s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106393948 (523h49m41.977476983s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106750467 (523h49m41.977833502s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107103769 (523h49m41.978186804s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107409381 (523h49m41.978492416s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107857575 (523h49m41.97894061s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.109167149 (523h49m41.980250184s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:43.813337727 (523h49m43.684420772s) [upload=459525]: Next returned nil segment Dec 27 11:18:43.813356433 (523h49m43.684439468s) [upload=459525]: Waiting for error group managing segments... Dec 27 11:18:44.097152137 (523h49m43.968235172s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.110645221 (523h49m43.981728256s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.111175320 (523h49m43.982258355s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.112899472 (523h49m43.983982518s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.114278359 (523h49m43.985361394s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.118051503 (523h49m43.989134538s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.121413054 (523h49m43.992496089s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.133847121 (523h49m44.004930156s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.304328606 (523h49m44.175411642s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.346564811 (523h49m44.217647846s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.489411776 (523h49m44.360494821s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.667495207 (523h49m44.538578242s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852043327 (523h49m44.723126372s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852061782 (523h49m44.723144817s) [seg_pos={0 0}, upload=459525]: Segment reached optimal threshold of 13 pieces. Dec 27 11:18:44.852259720 (523h49m44.723342755s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852274067 (523h49m44.723357102s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852499085 (523h49m44.72358212s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852505407 (523h49m44.723588452s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852601680 (523h49m44.723684745s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852622801 (523h49m44.723705846s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852654441 (523h49m44.723737496s) [seg_pos={0 0}, upload=459525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:18:44.852810758 (523h49m44.723893803s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:18:44.852818463 (523h49m44.723901509s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:18:45.138122149 (523h49m45.009205184s) [upload=459525]: Done uploading segments... Dec 27 11:18:45.138140855 (523h49m45.00922389s) [upload=459525]: Upload finished. err: <nil> Duration: 5.650886037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X] duration: 30.002915933s trace: https://jaeger.dev.storj.io/trace/51cbf38c3d4e9dfb FAIL (2025-12-27 11:19:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ) context deadline exceeded ======LOGS====== Dec 27 11:18:41.856771094 (523h49m41.727854129s) [upload=459525]: Starting upload Dec 27 11:18:41.856840466 (523h49m41.727923531s) [upload=459525]: Done starting upload Dec 27 11:18:41.856955636 (523h49m41.728038681s) [upload=459525]: Uploading segments... Dec 27 11:18:41.857101133 (523h49m41.728184208s) [upload=459525]: Got next segment. Inline: false Dec 27 11:18:41.857135749 (523h49m41.728218784s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:18:41.857141780 (523h49m41.728224825s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:18:42.102279323 (523h49m41.973362358s) [seg_pos={0 0}, upload=459525]: Begin upload segment... Dec 27 11:18:42.102319920 (523h49m41.973402955s) [seg_pos={0 0}, upload=459525]: Done begin upload segment. Dec 27 11:18:42.102355638 (523h49m41.973438673s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102363613 (523h49m41.973446648s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.102788452 (523h49m41.973871487s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103214694 (523h49m41.97429773s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103648410 (523h49m41.974731445s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.103984731 (523h49m41.975067766s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104248602 (523h49m41.975331648s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.104566207 (523h49m41.975649252s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106012291 (523h49m41.977095356s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106275342 (523h49m41.977358377s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106393948 (523h49m41.977476983s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.106750467 (523h49m41.977833502s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107103769 (523h49m41.978186804s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107409381 (523h49m41.978492416s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.107857575 (523h49m41.97894061s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:42.109167149 (523h49m41.980250184s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Uploading piece... Dec 27 11:18:43.813337727 (523h49m43.684420772s) [upload=459525]: Next returned nil segment Dec 27 11:18:43.813356433 (523h49m43.684439468s) [upload=459525]: Waiting for error group managing segments... Dec 27 11:18:44.097152137 (523h49m43.968235172s) [noise=true , address=145.79.227.17:4970 , piece_id=IUOHGWVAGVYPRGCXNXBFHFLLUIBJQMRYSXJOVX6OY2NPOKNJBCZA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.110645221 (523h49m43.981728256s) [noise=true , address=145.79.227.42:4123 , piece_id=GSBHP7DPNH2Z5Z3HJOVAJNNLB7P7CDGACTIMKFYIN5AS7OOWPRQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.111175320 (523h49m43.982258355s) [noise=true , address=145.79.227.50:4374 , piece_id=GLY2AAWTPZOFRZWY4FAOUOW3DVCMFBB2JIVF2SI3W4WC2LVVPKEA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.112899472 (523h49m43.983982518s) [noise=true , address=145.79.227.37:4566 , piece_id=RMPXDE2NANDWWZTW6RMROE2NXQHFA7TRYLTLX4AY2HC6AZIBAPAQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.114278359 (523h49m43.985361394s) [noise=true , address=145.79.227.39:4013 , piece_id=FZENGSS7ITD7HZNPJBDLQOTAGLZU5ABKI4PHOCKIPFU5KLNAB6HA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.118051503 (523h49m43.989134538s) [noise=true , address=145.79.227.51:4409 , piece_id=7YZG5IEBQB3AYKUQLVOG762PZGVECJND53IO6QAXXYWXRNHZ5LMQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.121413054 (523h49m43.992496089s) [noise=true , address=145.79.227.16:4909 , piece_id=L7QRZPLLOTMMKH5DTVL2QIWZTGF7STN2QEHDYQ7QAVJSZ6KBD4XQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.133847121 (523h49m44.004930156s) [noise=true , address=145.79.227.41:4085 , piece_id=UFWWCXZNEMQIOVYUU3OVYJFMZVDP654ZENLH4CES5TKOEWMBIDZQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.304328606 (523h49m44.175411642s) [noise=true , address=77.90.31.21:4880 , piece_id=C3FFY7OPT2MFVAV3VEBKW64FN7353ZQSCBTKXHZ6HY7SPQAUPOXQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.346564811 (523h49m44.217647846s) [noise=true , address=45.91.116.11:4118 , piece_id=GPYB65ZM6ENTMYLWBRGQCGJHE3I6BA7BZTXFDSECLSCHIURML5JQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.489411776 (523h49m44.360494821s) [noise=true , address=45.91.116.9:4264 , piece_id=3JBJJC57AQZ2EUUE5UYDL2S7T4IANCXZIWECMTXXBYFN3OJ4WJCQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.667495207 (523h49m44.538578242s) [noise=true , address=45.91.116.10:4084 , piece_id=EN66YV4I46BIIOJ3A3FWNVGD5SLCWGZH5YGO2SBBQJJH2CSQNHFQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852043327 (523h49m44.723126372s) [noise=true , address=45.91.116.12:4228 , piece_id=KZSB6QUNOFDLRVIFHI7N6BZ6WB564M2U7QGVRHWB4O6QYST37AQA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: <nil> Dec 27 11:18:44.852061782 (523h49m44.723144817s) [seg_pos={0 0}, upload=459525]: Segment reached optimal threshold of 13 pieces. Dec 27 11:18:44.852259720 (523h49m44.723342755s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852274067 (523h49m44.723357102s) [noise=true , address=145.79.227.33:4871 , piece_id=45QX33HTWVXOW2EPSWRIYHSAYI2YP4APXR7JBJDIWOD3527XQNUA, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852499085 (523h49m44.72358212s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852505407 (523h49m44.723588452s) [noise=true , address=45.91.116.14:4190 , piece_id=3H67WBT5EWP27MHCV64IVKBRYWEREPIX66KXPDXGFKOJQ7F2PUPQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852601680 (523h49m44.723684745s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 11:18:44.852622801 (523h49m44.723705846s) [noise=true , address=77.90.31.12:4589 , piece_id=XU3AUQRX2PXBJN3YFKFWO22FZJNDX5ZG7KYXH3QVBKY5A2RKDWBQ, seg_pos={0 0}, upload=459525]: Long tail cancellation due to optimal threshold Dec 27 11:18:44.852654441 (523h49m44.723737496s) [seg_pos={0 0}, upload=459525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:18:44.852810758 (523h49m44.723893803s) [upload=459525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:18:44.852818463 (523h49m44.723901509s) [upload=459525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:18:45.138122149 (523h49m45.009205184s) [upload=459525]: Done uploading segments... Dec 27 11:18:45.138140855 (523h49m45.00922389s) [upload=459525]: Upload finished. err: <nil> Duration: 30.002930261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.] duration: 5.956335817s trace: https://jaeger.dev.storj.io/trace/6755f39bc9ed606e FAIL (2025-12-27 11:18:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: KOSKLVP2POM26EVIZ4AATFAWRFHCGH53OMA5FNGESFVJS5O3HKDA) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: WDCFLXIEOYBEK5HBRXSEFH7KN62725HBD7BTNO3MS5S52L5KYUAA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 6NUJKJNQYYF2D7WF6HKSME34HTTT6XSFHCFXBI5VBQZXDZ6X5DNA) context deadline exceeded ======LOGS====== ent. Inline: false Dec 27 11:16:57.698555093 (523h47m57.569638138s) [upload=459505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:16:57.698562558 (523h47m57.569645593s) [upload=459505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:16:57.947742797 (523h47m57.818825832s) [seg_pos={0 0}, upload=459505]: Begin upload segment... Dec 27 11:16:57.947804475 (523h47m57.81888755s) [seg_pos={0 0}, upload=459505]: Done begin upload segment. Dec 27 11:16:57.947882994 (523h47m57.818966029s) [noise=true , address=45.91.116.5:4297 , piece_id=DEDXUUYXEWMM2HWBR3VB7ZCWRAQMP4532AYM5WZYKKCBB56E2CQQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.947924713 (523h47m57.819007749s) [noise=true , address=45.91.116.10:4083 , piece_id=TX7IH2H4NFDU56UN3F7PMJ7QET2X2T2WIGR3SKFCMMFUTPCZPE3A, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.948619977 (523h47m57.819703012s) [noise=true , address=45.91.116.11:4120 , piece_id=NQM2EYI7AKVEPJBEBPQOULAFIUOICKBCSWSV5TMD665BSKH7ZG3A, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.948897566 (523h47m57.819980611s) [noise=true , address=145.79.227.2:4113 , piece_id=VXYTRAGEBJCDCAZ3IS6OJ37H74RDLDFH7CEGXTX4KZN6OVGSJXGQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.950170380 (523h47m57.821253415s) [noise=true , address=145.79.227.28:4726 , piece_id=3MD7PWI3AFHKD7TQZHAS4FMIX4WH5XKL5WXKCYO6HEL6DR7FKPUA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.950185979 (523h47m57.821269014s) [noise=true , address=145.79.227.17:4971 , piece_id=67CH44WTO73I7TDZK35WGK3TUURG4KTG6MXRYYHEWP357OLQ4U6Q, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.950795019 (523h47m57.821878064s) [noise=true , address=45.91.116.6:4976 , piece_id=OUAF66L5SJXEKZCS7L2F2QIALKRHEORYRD43CG63SI74Z7R4TT7A, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.951437863 (523h47m57.822520898s) [noise=true , address=145.79.227.7:4294 , piece_id=WDCFLXIEOYBEK5HBRXSEFH7KN62725HBD7BTNO3MS5S52L5KYUAA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.952704034 (523h47m57.823787069s) [noise=true , address=145.79.227.43:4158 , piece_id=HCG2KLK6YU2LYNZSGIDSTZU5BRUHEUN6QA5FXAN52D3DFAKWWNRQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.954157271 (523h47m57.825240346s) [noise=true , address=77.90.31.12:4590 , piece_id=KFB7EHO6O5IT5SZS73JVXAE6L22HB47O4UV6QXZI4ZXZEY2YK37Q, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.954842526 (523h47m57.825925562s) [noise=true , address=145.79.227.48:4302 , piece_id=2XR5WRWLX4NFDBBMW5CC33CFFL25GVDXFQQ2ZF5G2VKETQX24UEA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.955132819 (523h47m57.826215864s) [noise=true , address=45.91.116.7:4010 , piece_id=6NUJKJNQYYF2D7WF6HKSME34HTTT6XSFHCFXBI5VBQZXDZ6X5DNA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.955469380 (523h47m57.826552425s) [noise=true , address=45.91.116.12:4228 , piece_id=L4SBD6AU3HK5MJ3GQFMXMCPCDKXIVF7KSLVZ6YAQWT47RH2JMYEQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.955677597 (523h47m57.826760632s) [noise=true , address=145.79.227.36:4518 , piece_id=3SQ2T526PDENFMH5X3SBKEJR5COKLVRZBWWXJSVZSTIE3RJHF7WA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.956228225 (523h47m57.82731127s) [noise=true , address=45.91.116.8:4045 , piece_id=KOSKLVP2POM26EVIZ4AATFAWRFHCGH53OMA5FNGESFVJS5O3HKDA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.957552577 (523h47m57.828635612s) [noise=true , address=145.79.227.38:4977 , piece_id=XRKI5YK4HPM7FA2TCPK6AZPN3XFGBQLIV2TVBIAUZKZ4E7AV4ZMA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:59.594856994 (523h47m59.465940029s) [upload=459505]: Next returned nil segment Dec 27 11:16:59.594870659 (523h47m59.465953695s) [upload=459505]: Waiting for error group managing segments... Dec 27 11:16:59.966000824 (523h47m59.837083869s) [noise=true , address=145.79.227.48:4302 , piece_id=2XR5WRWLX4NFDBBMW5CC33CFFL25GVDXFQQ2ZF5G2VKETQX24UEA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.966488763 (523h47m59.837571798s) [noise=true , address=145.79.227.7:4294 , piece_id=WDCFLXIEOYBEK5HBRXSEFH7KN62725HBD7BTNO3MS5S52L5KYUAA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.976977295 (523h47m59.84806034s) [noise=true , address=145.79.227.2:4113 , piece_id=VXYTRAGEBJCDCAZ3IS6OJ37H74RDLDFH7CEGXTX4KZN6OVGSJXGQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.978429220 (523h47m59.849512285s) [noise=true , address=145.79.227.17:4971 , piece_id=67CH44WTO73I7TDZK35WGK3TUURG4KTG6MXRYYHEWP357OLQ4U6Q, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.982795243 (523h47m59.853878278s) [noise=true , address=145.79.227.36:4518 , piece_id=3SQ2T526PDENFMH5X3SBKEJR5COKLVRZBWWXJSVZSTIE3RJHF7WA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.982890124 (523h47m59.853973159s) [noise=true , address=145.79.227.28:4726 , piece_id=3MD7PWI3AFHKD7TQZHAS4FMIX4WH5XKL5WXKCYO6HEL6DR7FKPUA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.983284074 (523h47m59.854367129s) [noise=true , address=145.79.227.43:4158 , piece_id=HCG2KLK6YU2LYNZSGIDSTZU5BRUHEUN6QA5FXAN52D3DFAKWWNRQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.984171384 (523h47m59.855254429s) [noise=true , address=145.79.227.38:4977 , piece_id=XRKI5YK4HPM7FA2TCPK6AZPN3XFGBQLIV2TVBIAUZKZ4E7AV4ZMA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.207905671 (523h48m0.078988717s) [noise=true , address=45.91.116.8:4045 , piece_id=KOSKLVP2POM26EVIZ4AATFAWRFHCGH53OMA5FNGESFVJS5O3HKDA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.239878664 (523h48m0.110961709s) [noise=true , address=45.91.116.7:4010 , piece_id=6NUJKJNQYYF2D7WF6HKSME34HTTT6XSFHCFXBI5VBQZXDZ6X5DNA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.320356276 (523h48m0.191439322s) [noise=true , address=45.91.116.12:4228 , piece_id=L4SBD6AU3HK5MJ3GQFMXMCPCDKXIVF7KSLVZ6YAQWT47RH2JMYEQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.349141088 (523h48m0.220224123s) [noise=true , address=45.91.116.5:4297 , piece_id=DEDXUUYXEWMM2HWBR3VB7ZCWRAQMP4532AYM5WZYKKCBB56E2CQQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.549894606 (523h48m0.420977651s) [noise=true , address=45.91.116.6:4976 , piece_id=OUAF66L5SJXEKZCS7L2F2QIALKRHEORYRD43CG63SI74Z7R4TT7A, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.549926908 (523h48m0.421009953s) [seg_pos={0 0}, upload=459505]: Segment reached optimal threshold of 13 pieces. Dec 27 11:17:00.550156115 (523h48m0.42123915s) [noise=true , address=45.91.116.10:4083 , piece_id=TX7IH2H4NFDU56UN3F7PMJ7QET2X2T2WIGR3SKFCMMFUTPCZPE3A, seg_pos={0 0}, upload=459505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:60352->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:60352->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60352->45.91.116.10:4083: use of closed network connection Dec 27 11:17:00.550164280 (523h48m0.421247325s) [noise=true , address=45.91.116.10:4083 , piece_id=TX7IH2H4NFDU56UN3F7PMJ7QET2X2T2WIGR3SKFCMMFUTPCZPE3A, seg_pos={0 0}, upload=459505]: Long tail cancellation due to optimal threshold Dec 27 11:17:00.550194037 (523h48m0.421277082s) [noise=true , address=45.91.116.11:4120 , piece_id=NQM2EYI7AKVEPJBEBPQOULAFIUOICKBCSWSV5TMD665BSKH7ZG3A, seg_pos={0 0}, upload=459505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:44052->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:44052->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44052->45.91.116.11:4120: use of closed network connection Dec 27 11:17:00.550204326 (523h48m0.421287361s) [noise=true , address=45.91.116.11:4120 , piece_id=NQM2EYI7AKVEPJBEBPQOULAFIUOICKBCSWSV5TMD665BSKH7ZG3A, seg_pos={0 0}, upload=459505]: Long tail cancellation due to optimal threshold Dec 27 11:17:00.550462869 (523h48m0.421545904s) [noise=true , address=77.90.31.12:4590 , piece_id=KFB7EHO6O5IT5SZS73JVXAE6L22HB47O4UV6QXZI4ZXZEY2YK37Q, seg_pos={0 0}, upload=459505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:37876->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 11:17:00.550471305 (523h48m0.42155435s) [noise=true , address=77.90.31.12:4590 , piece_id=KFB7EHO6O5IT5SZS73JVXAE6L22HB47O4UV6QXZI4ZXZEY2YK37Q, seg_pos={0 0}, upload=459505]: Long tail cancellation due to optimal threshold Dec 27 11:17:00.550485471 (523h48m0.421568516s) [seg_pos={0 0}, upload=459505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:17:00.550608195 (523h48m0.421691231s) [upload=459505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:17:00.550613636 (523h48m0.421696671s) [upload=459505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:17:00.847370701 (523h48m0.718453737s) [upload=459505]: Done uploading segments... Dec 27 11:17:00.847429583 (523h48m0.718512618s) [upload=459505]: Upload finished. err: <nil> Duration: 5.9563784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X] duration: 30.003839674s trace: https://jaeger.dev.storj.io/trace/6755f39bc9ed606e FAIL (2025-12-27 11:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: KOSKLVP2POM26EVIZ4AATFAWRFHCGH53OMA5FNGESFVJS5O3HKDA) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: WDCFLXIEOYBEK5HBRXSEFH7KN62725HBD7BTNO3MS5S52L5KYUAA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 6NUJKJNQYYF2D7WF6HKSME34HTTT6XSFHCFXBI5VBQZXDZ6X5DNA) context deadline exceeded ======LOGS====== ent. Inline: false Dec 27 11:16:57.698555093 (523h47m57.569638138s) [upload=459505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:16:57.698562558 (523h47m57.569645593s) [upload=459505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:16:57.947742797 (523h47m57.818825832s) [seg_pos={0 0}, upload=459505]: Begin upload segment... Dec 27 11:16:57.947804475 (523h47m57.81888755s) [seg_pos={0 0}, upload=459505]: Done begin upload segment. Dec 27 11:16:57.947882994 (523h47m57.818966029s) [noise=true , address=45.91.116.5:4297 , piece_id=DEDXUUYXEWMM2HWBR3VB7ZCWRAQMP4532AYM5WZYKKCBB56E2CQQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.947924713 (523h47m57.819007749s) [noise=true , address=45.91.116.10:4083 , piece_id=TX7IH2H4NFDU56UN3F7PMJ7QET2X2T2WIGR3SKFCMMFUTPCZPE3A, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.948619977 (523h47m57.819703012s) [noise=true , address=45.91.116.11:4120 , piece_id=NQM2EYI7AKVEPJBEBPQOULAFIUOICKBCSWSV5TMD665BSKH7ZG3A, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.948897566 (523h47m57.819980611s) [noise=true , address=145.79.227.2:4113 , piece_id=VXYTRAGEBJCDCAZ3IS6OJ37H74RDLDFH7CEGXTX4KZN6OVGSJXGQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.950170380 (523h47m57.821253415s) [noise=true , address=145.79.227.28:4726 , piece_id=3MD7PWI3AFHKD7TQZHAS4FMIX4WH5XKL5WXKCYO6HEL6DR7FKPUA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.950185979 (523h47m57.821269014s) [noise=true , address=145.79.227.17:4971 , piece_id=67CH44WTO73I7TDZK35WGK3TUURG4KTG6MXRYYHEWP357OLQ4U6Q, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.950795019 (523h47m57.821878064s) [noise=true , address=45.91.116.6:4976 , piece_id=OUAF66L5SJXEKZCS7L2F2QIALKRHEORYRD43CG63SI74Z7R4TT7A, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.951437863 (523h47m57.822520898s) [noise=true , address=145.79.227.7:4294 , piece_id=WDCFLXIEOYBEK5HBRXSEFH7KN62725HBD7BTNO3MS5S52L5KYUAA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.952704034 (523h47m57.823787069s) [noise=true , address=145.79.227.43:4158 , piece_id=HCG2KLK6YU2LYNZSGIDSTZU5BRUHEUN6QA5FXAN52D3DFAKWWNRQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.954157271 (523h47m57.825240346s) [noise=true , address=77.90.31.12:4590 , piece_id=KFB7EHO6O5IT5SZS73JVXAE6L22HB47O4UV6QXZI4ZXZEY2YK37Q, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.954842526 (523h47m57.825925562s) [noise=true , address=145.79.227.48:4302 , piece_id=2XR5WRWLX4NFDBBMW5CC33CFFL25GVDXFQQ2ZF5G2VKETQX24UEA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.955132819 (523h47m57.826215864s) [noise=true , address=45.91.116.7:4010 , piece_id=6NUJKJNQYYF2D7WF6HKSME34HTTT6XSFHCFXBI5VBQZXDZ6X5DNA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.955469380 (523h47m57.826552425s) [noise=true , address=45.91.116.12:4228 , piece_id=L4SBD6AU3HK5MJ3GQFMXMCPCDKXIVF7KSLVZ6YAQWT47RH2JMYEQ, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.955677597 (523h47m57.826760632s) [noise=true , address=145.79.227.36:4518 , piece_id=3SQ2T526PDENFMH5X3SBKEJR5COKLVRZBWWXJSVZSTIE3RJHF7WA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.956228225 (523h47m57.82731127s) [noise=true , address=45.91.116.8:4045 , piece_id=KOSKLVP2POM26EVIZ4AATFAWRFHCGH53OMA5FNGESFVJS5O3HKDA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:57.957552577 (523h47m57.828635612s) [noise=true , address=145.79.227.38:4977 , piece_id=XRKI5YK4HPM7FA2TCPK6AZPN3XFGBQLIV2TVBIAUZKZ4E7AV4ZMA, seg_pos={0 0}, upload=459505]: Uploading piece... Dec 27 11:16:59.594856994 (523h47m59.465940029s) [upload=459505]: Next returned nil segment Dec 27 11:16:59.594870659 (523h47m59.465953695s) [upload=459505]: Waiting for error group managing segments... Dec 27 11:16:59.966000824 (523h47m59.837083869s) [noise=true , address=145.79.227.48:4302 , piece_id=2XR5WRWLX4NFDBBMW5CC33CFFL25GVDXFQQ2ZF5G2VKETQX24UEA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.966488763 (523h47m59.837571798s) [noise=true , address=145.79.227.7:4294 , piece_id=WDCFLXIEOYBEK5HBRXSEFH7KN62725HBD7BTNO3MS5S52L5KYUAA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.976977295 (523h47m59.84806034s) [noise=true , address=145.79.227.2:4113 , piece_id=VXYTRAGEBJCDCAZ3IS6OJ37H74RDLDFH7CEGXTX4KZN6OVGSJXGQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.978429220 (523h47m59.849512285s) [noise=true , address=145.79.227.17:4971 , piece_id=67CH44WTO73I7TDZK35WGK3TUURG4KTG6MXRYYHEWP357OLQ4U6Q, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.982795243 (523h47m59.853878278s) [noise=true , address=145.79.227.36:4518 , piece_id=3SQ2T526PDENFMH5X3SBKEJR5COKLVRZBWWXJSVZSTIE3RJHF7WA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.982890124 (523h47m59.853973159s) [noise=true , address=145.79.227.28:4726 , piece_id=3MD7PWI3AFHKD7TQZHAS4FMIX4WH5XKL5WXKCYO6HEL6DR7FKPUA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.983284074 (523h47m59.854367129s) [noise=true , address=145.79.227.43:4158 , piece_id=HCG2KLK6YU2LYNZSGIDSTZU5BRUHEUN6QA5FXAN52D3DFAKWWNRQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:16:59.984171384 (523h47m59.855254429s) [noise=true , address=145.79.227.38:4977 , piece_id=XRKI5YK4HPM7FA2TCPK6AZPN3XFGBQLIV2TVBIAUZKZ4E7AV4ZMA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.207905671 (523h48m0.078988717s) [noise=true , address=45.91.116.8:4045 , piece_id=KOSKLVP2POM26EVIZ4AATFAWRFHCGH53OMA5FNGESFVJS5O3HKDA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.239878664 (523h48m0.110961709s) [noise=true , address=45.91.116.7:4010 , piece_id=6NUJKJNQYYF2D7WF6HKSME34HTTT6XSFHCFXBI5VBQZXDZ6X5DNA, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.320356276 (523h48m0.191439322s) [noise=true , address=45.91.116.12:4228 , piece_id=L4SBD6AU3HK5MJ3GQFMXMCPCDKXIVF7KSLVZ6YAQWT47RH2JMYEQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.349141088 (523h48m0.220224123s) [noise=true , address=45.91.116.5:4297 , piece_id=DEDXUUYXEWMM2HWBR3VB7ZCWRAQMP4532AYM5WZYKKCBB56E2CQQ, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.549894606 (523h48m0.420977651s) [noise=true , address=45.91.116.6:4976 , piece_id=OUAF66L5SJXEKZCS7L2F2QIALKRHEORYRD43CG63SI74Z7R4TT7A, seg_pos={0 0}, upload=459505]: Done uploading piece. err: <nil> Dec 27 11:17:00.549926908 (523h48m0.421009953s) [seg_pos={0 0}, upload=459505]: Segment reached optimal threshold of 13 pieces. Dec 27 11:17:00.550156115 (523h48m0.42123915s) [noise=true , address=45.91.116.10:4083 , piece_id=TX7IH2H4NFDU56UN3F7PMJ7QET2X2T2WIGR3SKFCMMFUTPCZPE3A, seg_pos={0 0}, upload=459505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:60352->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:60352->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60352->45.91.116.10:4083: use of closed network connection Dec 27 11:17:00.550164280 (523h48m0.421247325s) [noise=true , address=45.91.116.10:4083 , piece_id=TX7IH2H4NFDU56UN3F7PMJ7QET2X2T2WIGR3SKFCMMFUTPCZPE3A, seg_pos={0 0}, upload=459505]: Long tail cancellation due to optimal threshold Dec 27 11:17:00.550194037 (523h48m0.421277082s) [noise=true , address=45.91.116.11:4120 , piece_id=NQM2EYI7AKVEPJBEBPQOULAFIUOICKBCSWSV5TMD665BSKH7ZG3A, seg_pos={0 0}, upload=459505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:44052->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:44052->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44052->45.91.116.11:4120: use of closed network connection Dec 27 11:17:00.550204326 (523h48m0.421287361s) [noise=true , address=45.91.116.11:4120 , piece_id=NQM2EYI7AKVEPJBEBPQOULAFIUOICKBCSWSV5TMD665BSKH7ZG3A, seg_pos={0 0}, upload=459505]: Long tail cancellation due to optimal threshold Dec 27 11:17:00.550462869 (523h48m0.421545904s) [noise=true , address=77.90.31.12:4590 , piece_id=KFB7EHO6O5IT5SZS73JVXAE6L22HB47O4UV6QXZI4ZXZEY2YK37Q, seg_pos={0 0}, upload=459505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:37876->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 11:17:00.550471305 (523h48m0.42155435s) [noise=true , address=77.90.31.12:4590 , piece_id=KFB7EHO6O5IT5SZS73JVXAE6L22HB47O4UV6QXZI4ZXZEY2YK37Q, seg_pos={0 0}, upload=459505]: Long tail cancellation due to optimal threshold Dec 27 11:17:00.550485471 (523h48m0.421568516s) [seg_pos={0 0}, upload=459505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:17:00.550608195 (523h48m0.421691231s) [upload=459505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:17:00.550613636 (523h48m0.421696671s) [upload=459505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:17:00.847370701 (523h48m0.718453737s) [upload=459505]: Done uploading segments... Dec 27 11:17:00.847429583 (523h48m0.718512618s) [upload=459505]: Upload finished. err: <nil> Duration: 30.003865343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.] duration: 5.904569145s trace: https://jaeger.dev.storj.io/trace/64902a1ee8af3a86 FAIL (2025-12-27 11:15:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 11:15:18.726008953 (523h46m18.597091998s) [upload=459479]: Starting upload Dec 27 11:15:18.726087111 (523h46m18.597170167s) [upload=459479]: Done starting upload Dec 27 11:15:18.726214414 (523h46m18.597297449s) [upload=459479]: Uploading segments... Dec 27 11:15:18.726339041 (523h46m18.597422076s) [upload=459479]: Got next segment. Inline: false Dec 27 11:15:18.726374609 (523h46m18.597457654s) [upload=459479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:15:18.726381431 (523h46m18.597464466s) [upload=459479]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:15:18.970274625 (523h46m18.84135767s) [seg_pos={0 0}, upload=459479]: Begin upload segment... Dec 27 11:15:18.970337104 (523h46m18.841420139s) [seg_pos={0 0}, upload=459479]: Done begin upload segment. Dec 27 11:15:18.970355188 (523h46m18.841438233s) [noise=true , address=145.79.227.41:4087 , piece_id=P352FR2ZSPX2UHCU6BGHGTVZ4HDLPBFZB5LUBUAVQKLPMQYJQVFA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.970373012 (523h46m18.841456058s) [noise=true , address=145.79.227.35:4482 , piece_id=444EY272ANTV4UEOIDTXDVQQ76QHHOSIWQO7A6EGPKCKII2K6RKA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.970847166 (523h46m18.841930201s) [noise=true , address=45.91.116.12:4228 , piece_id=47RJ67BJ5R36A7H564NB4IJCE3KEHNRDHWCISUCQIBD6BFOXXBLA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.971016708 (523h46m18.842099743s) [noise=true , address=145.79.227.37:4566 , piece_id=WAWFI7XY2PCCUX43CUYU6XU347GQNR3ZRXP76Z7KBSDBE5SAXOYA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.971397453 (523h46m18.842480488s) [noise=true , address=145.79.227.8:4331 , piece_id=6WITZWTNVUSXPEVI6PVUQQ4QYIHEIMNCG4DBBRSOSX5XDVHOWSJQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.971769842 (523h46m18.842852877s) [noise=true , address=145.79.227.42:4121 , piece_id=GRGOGFAWGNZ3AVLMIOVLDANQCLW4XMIC5QVB5C7V3QSGR37DUBAA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.972133735 (523h46m18.84321678s) [noise=true , address=145.79.227.49:4339 , piece_id=LH7AJKWCO2GYNCEDRIX7PVPHK2B5GB7TQA45323UFTH6SK4XPZZA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.972519340 (523h46m18.843602375s) [noise=true , address=77.90.31.16:4735 , piece_id=OZMENFTEZLUXRQPY2LFVFRNIAV75MNGIYPVO2P7ODBTJBQMNXLYQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.975051160 (523h46m18.846134205s) [noise=true , address=45.91.116.11:4119 , piece_id=ES4C7UJTWL5FONBJTRIFUP77J4U54P4NJ7D7ZOPQGPGWR2XZYLZA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.976052467 (523h46m18.847135502s) [noise=true , address=145.79.227.39:4014 , piece_id=JYFFLZGIXZHNDUOEW7XFVMGJLI72WO73ZQRLCO65J2JZ5PEI6OEA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.977080806 (523h46m18.848163851s) [noise=true , address=145.79.227.51:4409 , piece_id=PIJUFEWVRLN365P6JM7WDQHQMEAQXUNJ6OQMRR4MOYVCT7SF34QQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.979045456 (523h46m18.850128491s) [noise=true , address=145.79.227.16:4912 , piece_id=5NTBUGQEH7QPMVNOQBOT5WY4CG72HZD3HCR5R56RJ2JX6QMMTFSA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.980487512 (523h46m18.851570547s) [noise=true , address=45.91.116.13:4155 , piece_id=XN6XS2O5WVXEL5IOL4YRJ35ISEDT7Y43ZPST7LWEZ3BUJYALMLTQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.981945298 (523h46m18.853028333s) [noise=true , address=45.91.116.8:4045 , piece_id=GZQDN6XJRAFIHAZIKTPQSM7W3HHMJ2SLCZLQTL4LBEMUZGK6ZCIA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.983525838 (523h46m18.854608883s) [noise=true , address=145.79.227.15:4873 , piece_id=YAQ3NJDZMG2DUTO3JY7BXV75HQEF5FQJGNFZ4IIKLCKACC6RQDNQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.986962803 (523h46m18.858045838s) [noise=true , address=45.91.116.4:4939 , piece_id=LPXMQO5OGOAVHBQ6YFLGA55JAFOF6UX2UDZPGZLZIW6PRX3OR7UA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:20.694415201 (523h46m20.565498236s) [upload=459479]: Next returned nil segment Dec 27 11:15:20.694463985 (523h46m20.56554703s) [upload=459479]: Waiting for error group managing segments... Dec 27 11:15:20.980114841 (523h46m20.851197887s) [noise=true , address=145.79.227.8:4331 , piece_id=6WITZWTNVUSXPEVI6PVUQQ4QYIHEIMNCG4DBBRSOSX5XDVHOWSJQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.980308661 (523h46m20.851391696s) [noise=true , address=145.79.227.37:4566 , piece_id=WAWFI7XY2PCCUX43CUYU6XU347GQNR3ZRXP76Z7KBSDBE5SAXOYA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.980547555 (523h46m20.851649016s) [noise=true , address=145.79.227.41:4087 , piece_id=P352FR2ZSPX2UHCU6BGHGTVZ4HDLPBFZB5LUBUAVQKLPMQYJQVFA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.981481223 (523h46m20.852564268s) [noise=true , address=145.79.227.49:4339 , piece_id=LH7AJKWCO2GYNCEDRIX7PVPHK2B5GB7TQA45323UFTH6SK4XPZZA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.992908011 (523h46m20.863991046s) [noise=true , address=145.79.227.39:4014 , piece_id=JYFFLZGIXZHNDUOEW7XFVMGJLI72WO73ZQRLCO65J2JZ5PEI6OEA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.994426373 (523h46m20.865509418s) [noise=true , address=145.79.227.42:4121 , piece_id=GRGOGFAWGNZ3AVLMIOVLDANQCLW4XMIC5QVB5C7V3QSGR37DUBAA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.997637878 (523h46m20.868720923s) [noise=true , address=145.79.227.16:4912 , piece_id=5NTBUGQEH7QPMVNOQBOT5WY4CG72HZD3HCR5R56RJ2JX6QMMTFSA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.998244494 (523h46m20.869327529s) [noise=true , address=145.79.227.51:4409 , piece_id=PIJUFEWVRLN365P6JM7WDQHQMEAQXUNJ6OQMRR4MOYVCT7SF34QQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.003131519 (523h46m20.874214554s) [noise=true , address=145.79.227.35:4482 , piece_id=444EY272ANTV4UEOIDTXDVQQ76QHHOSIWQO7A6EGPKCKII2K6RKA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.013860248 (523h46m20.884943293s) [noise=true , address=145.79.227.15:4873 , piece_id=YAQ3NJDZMG2DUTO3JY7BXV75HQEF5FQJGNFZ4IIKLCKACC6RQDNQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.159762371 (523h46m21.030845406s) [noise=true , address=77.90.31.16:4735 , piece_id=OZMENFTEZLUXRQPY2LFVFRNIAV75MNGIYPVO2P7ODBTJBQMNXLYQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.235878518 (523h46m21.106961553s) [noise=true , address=45.91.116.8:4045 , piece_id=GZQDN6XJRAFIHAZIKTPQSM7W3HHMJ2SLCZLQTL4LBEMUZGK6ZCIA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.296964206 (523h46m21.168047241s) [noise=true , address=45.91.116.13:4155 , piece_id=XN6XS2O5WVXEL5IOL4YRJ35ISEDT7Y43ZPST7LWEZ3BUJYALMLTQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.296984465 (523h46m21.1680675s) [seg_pos={0 0}, upload=459479]: Segment reached optimal threshold of 13 pieces. Dec 27 11:15:21.297184687 (523h46m21.168267722s) [noise=true , address=45.91.116.4:4939 , piece_id=LPXMQO5OGOAVHBQ6YFLGA55JAFOF6UX2UDZPGZLZIW6PRX3OR7UA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:35366->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:35366->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35366->45.91.116.4:4939: use of closed network connection Dec 27 11:15:21.297192762 (523h46m21.168275807s) [noise=true , address=45.91.116.4:4939 , piece_id=LPXMQO5OGOAVHBQ6YFLGA55JAFOF6UX2UDZPGZLZIW6PRX3OR7UA, seg_pos={0 0}, upload=459479]: Long tail cancellation due to optimal threshold Dec 27 11:15:21.297403323 (523h46m21.168486368s) [noise=true , address=45.91.116.12:4228 , piece_id=47RJ67BJ5R36A7H564NB4IJCE3KEHNRDHWCISUCQIBD6BFOXXBLA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 11:15:21.297410566 (523h46m21.168493601s) [noise=true , address=45.91.116.12:4228 , piece_id=47RJ67BJ5R36A7H564NB4IJCE3KEHNRDHWCISUCQIBD6BFOXXBLA, seg_pos={0 0}, upload=459479]: Long tail cancellation due to optimal threshold Dec 27 11:15:21.297425304 (523h46m21.168508339s) [noise=true , address=45.91.116.11:4119 , piece_id=ES4C7UJTWL5FONBJTRIFUP77J4U54P4NJ7D7ZOPQGPGWR2XZYLZA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 11:15:21.297431487 (523h46m21.168514522s) [noise=true , address=45.91.116.11:4119 , piece_id=ES4C7UJTWL5FONBJTRIFUP77J4U54P4NJ7D7ZOPQGPGWR2XZYLZA, seg_pos={0 0}, upload=459479]: Long tail cancellation due to optimal threshold Dec 27 11:15:21.297445553 (523h46m21.168528588s) [seg_pos={0 0}, upload=459479]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:15:21.297511849 (523h46m21.168594894s) [upload=459479]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:15:21.297516348 (523h46m21.168599383s) [upload=459479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:15:21.588709038 (523h46m21.459792073s) [upload=459479]: Done uploading segments... Dec 27 11:15:21.588743804 (523h46m21.459826839s) [upload=459479]: Upload finished. err: <nil> Duration: 5.904599725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X] duration: 30.003076631s trace: https://jaeger.dev.storj.io/trace/64902a1ee8af3a86 FAIL (2025-12-27 11:15:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 11:15:18.726008953 (523h46m18.597091998s) [upload=459479]: Starting upload Dec 27 11:15:18.726087111 (523h46m18.597170167s) [upload=459479]: Done starting upload Dec 27 11:15:18.726214414 (523h46m18.597297449s) [upload=459479]: Uploading segments... Dec 27 11:15:18.726339041 (523h46m18.597422076s) [upload=459479]: Got next segment. Inline: false Dec 27 11:15:18.726374609 (523h46m18.597457654s) [upload=459479]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:15:18.726381431 (523h46m18.597464466s) [upload=459479]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:15:18.970274625 (523h46m18.84135767s) [seg_pos={0 0}, upload=459479]: Begin upload segment... Dec 27 11:15:18.970337104 (523h46m18.841420139s) [seg_pos={0 0}, upload=459479]: Done begin upload segment. Dec 27 11:15:18.970355188 (523h46m18.841438233s) [noise=true , address=145.79.227.41:4087 , piece_id=P352FR2ZSPX2UHCU6BGHGTVZ4HDLPBFZB5LUBUAVQKLPMQYJQVFA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.970373012 (523h46m18.841456058s) [noise=true , address=145.79.227.35:4482 , piece_id=444EY272ANTV4UEOIDTXDVQQ76QHHOSIWQO7A6EGPKCKII2K6RKA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.970847166 (523h46m18.841930201s) [noise=true , address=45.91.116.12:4228 , piece_id=47RJ67BJ5R36A7H564NB4IJCE3KEHNRDHWCISUCQIBD6BFOXXBLA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.971016708 (523h46m18.842099743s) [noise=true , address=145.79.227.37:4566 , piece_id=WAWFI7XY2PCCUX43CUYU6XU347GQNR3ZRXP76Z7KBSDBE5SAXOYA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.971397453 (523h46m18.842480488s) [noise=true , address=145.79.227.8:4331 , piece_id=6WITZWTNVUSXPEVI6PVUQQ4QYIHEIMNCG4DBBRSOSX5XDVHOWSJQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.971769842 (523h46m18.842852877s) [noise=true , address=145.79.227.42:4121 , piece_id=GRGOGFAWGNZ3AVLMIOVLDANQCLW4XMIC5QVB5C7V3QSGR37DUBAA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.972133735 (523h46m18.84321678s) [noise=true , address=145.79.227.49:4339 , piece_id=LH7AJKWCO2GYNCEDRIX7PVPHK2B5GB7TQA45323UFTH6SK4XPZZA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.972519340 (523h46m18.843602375s) [noise=true , address=77.90.31.16:4735 , piece_id=OZMENFTEZLUXRQPY2LFVFRNIAV75MNGIYPVO2P7ODBTJBQMNXLYQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.975051160 (523h46m18.846134205s) [noise=true , address=45.91.116.11:4119 , piece_id=ES4C7UJTWL5FONBJTRIFUP77J4U54P4NJ7D7ZOPQGPGWR2XZYLZA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.976052467 (523h46m18.847135502s) [noise=true , address=145.79.227.39:4014 , piece_id=JYFFLZGIXZHNDUOEW7XFVMGJLI72WO73ZQRLCO65J2JZ5PEI6OEA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.977080806 (523h46m18.848163851s) [noise=true , address=145.79.227.51:4409 , piece_id=PIJUFEWVRLN365P6JM7WDQHQMEAQXUNJ6OQMRR4MOYVCT7SF34QQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.979045456 (523h46m18.850128491s) [noise=true , address=145.79.227.16:4912 , piece_id=5NTBUGQEH7QPMVNOQBOT5WY4CG72HZD3HCR5R56RJ2JX6QMMTFSA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.980487512 (523h46m18.851570547s) [noise=true , address=45.91.116.13:4155 , piece_id=XN6XS2O5WVXEL5IOL4YRJ35ISEDT7Y43ZPST7LWEZ3BUJYALMLTQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.981945298 (523h46m18.853028333s) [noise=true , address=45.91.116.8:4045 , piece_id=GZQDN6XJRAFIHAZIKTPQSM7W3HHMJ2SLCZLQTL4LBEMUZGK6ZCIA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.983525838 (523h46m18.854608883s) [noise=true , address=145.79.227.15:4873 , piece_id=YAQ3NJDZMG2DUTO3JY7BXV75HQEF5FQJGNFZ4IIKLCKACC6RQDNQ, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:18.986962803 (523h46m18.858045838s) [noise=true , address=45.91.116.4:4939 , piece_id=LPXMQO5OGOAVHBQ6YFLGA55JAFOF6UX2UDZPGZLZIW6PRX3OR7UA, seg_pos={0 0}, upload=459479]: Uploading piece... Dec 27 11:15:20.694415201 (523h46m20.565498236s) [upload=459479]: Next returned nil segment Dec 27 11:15:20.694463985 (523h46m20.56554703s) [upload=459479]: Waiting for error group managing segments... Dec 27 11:15:20.980114841 (523h46m20.851197887s) [noise=true , address=145.79.227.8:4331 , piece_id=6WITZWTNVUSXPEVI6PVUQQ4QYIHEIMNCG4DBBRSOSX5XDVHOWSJQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.980308661 (523h46m20.851391696s) [noise=true , address=145.79.227.37:4566 , piece_id=WAWFI7XY2PCCUX43CUYU6XU347GQNR3ZRXP76Z7KBSDBE5SAXOYA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.980547555 (523h46m20.851649016s) [noise=true , address=145.79.227.41:4087 , piece_id=P352FR2ZSPX2UHCU6BGHGTVZ4HDLPBFZB5LUBUAVQKLPMQYJQVFA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.981481223 (523h46m20.852564268s) [noise=true , address=145.79.227.49:4339 , piece_id=LH7AJKWCO2GYNCEDRIX7PVPHK2B5GB7TQA45323UFTH6SK4XPZZA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.992908011 (523h46m20.863991046s) [noise=true , address=145.79.227.39:4014 , piece_id=JYFFLZGIXZHNDUOEW7XFVMGJLI72WO73ZQRLCO65J2JZ5PEI6OEA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.994426373 (523h46m20.865509418s) [noise=true , address=145.79.227.42:4121 , piece_id=GRGOGFAWGNZ3AVLMIOVLDANQCLW4XMIC5QVB5C7V3QSGR37DUBAA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.997637878 (523h46m20.868720923s) [noise=true , address=145.79.227.16:4912 , piece_id=5NTBUGQEH7QPMVNOQBOT5WY4CG72HZD3HCR5R56RJ2JX6QMMTFSA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:20.998244494 (523h46m20.869327529s) [noise=true , address=145.79.227.51:4409 , piece_id=PIJUFEWVRLN365P6JM7WDQHQMEAQXUNJ6OQMRR4MOYVCT7SF34QQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.003131519 (523h46m20.874214554s) [noise=true , address=145.79.227.35:4482 , piece_id=444EY272ANTV4UEOIDTXDVQQ76QHHOSIWQO7A6EGPKCKII2K6RKA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.013860248 (523h46m20.884943293s) [noise=true , address=145.79.227.15:4873 , piece_id=YAQ3NJDZMG2DUTO3JY7BXV75HQEF5FQJGNFZ4IIKLCKACC6RQDNQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.159762371 (523h46m21.030845406s) [noise=true , address=77.90.31.16:4735 , piece_id=OZMENFTEZLUXRQPY2LFVFRNIAV75MNGIYPVO2P7ODBTJBQMNXLYQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.235878518 (523h46m21.106961553s) [noise=true , address=45.91.116.8:4045 , piece_id=GZQDN6XJRAFIHAZIKTPQSM7W3HHMJ2SLCZLQTL4LBEMUZGK6ZCIA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.296964206 (523h46m21.168047241s) [noise=true , address=45.91.116.13:4155 , piece_id=XN6XS2O5WVXEL5IOL4YRJ35ISEDT7Y43ZPST7LWEZ3BUJYALMLTQ, seg_pos={0 0}, upload=459479]: Done uploading piece. err: <nil> Dec 27 11:15:21.296984465 (523h46m21.1680675s) [seg_pos={0 0}, upload=459479]: Segment reached optimal threshold of 13 pieces. Dec 27 11:15:21.297184687 (523h46m21.168267722s) [noise=true , address=45.91.116.4:4939 , piece_id=LPXMQO5OGOAVHBQ6YFLGA55JAFOF6UX2UDZPGZLZIW6PRX3OR7UA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:35366->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:35366->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35366->45.91.116.4:4939: use of closed network connection Dec 27 11:15:21.297192762 (523h46m21.168275807s) [noise=true , address=45.91.116.4:4939 , piece_id=LPXMQO5OGOAVHBQ6YFLGA55JAFOF6UX2UDZPGZLZIW6PRX3OR7UA, seg_pos={0 0}, upload=459479]: Long tail cancellation due to optimal threshold Dec 27 11:15:21.297403323 (523h46m21.168486368s) [noise=true , address=45.91.116.12:4228 , piece_id=47RJ67BJ5R36A7H564NB4IJCE3KEHNRDHWCISUCQIBD6BFOXXBLA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 11:15:21.297410566 (523h46m21.168493601s) [noise=true , address=45.91.116.12:4228 , piece_id=47RJ67BJ5R36A7H564NB4IJCE3KEHNRDHWCISUCQIBD6BFOXXBLA, seg_pos={0 0}, upload=459479]: Long tail cancellation due to optimal threshold Dec 27 11:15:21.297425304 (523h46m21.168508339s) [noise=true , address=45.91.116.11:4119 , piece_id=ES4C7UJTWL5FONBJTRIFUP77J4U54P4NJ7D7ZOPQGPGWR2XZYLZA, seg_pos={0 0}, upload=459479]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 27 11:15:21.297431487 (523h46m21.168514522s) [noise=true , address=45.91.116.11:4119 , piece_id=ES4C7UJTWL5FONBJTRIFUP77J4U54P4NJ7D7ZOPQGPGWR2XZYLZA, seg_pos={0 0}, upload=459479]: Long tail cancellation due to optimal threshold Dec 27 11:15:21.297445553 (523h46m21.168528588s) [seg_pos={0 0}, upload=459479]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:15:21.297511849 (523h46m21.168594894s) [upload=459479]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:15:21.297516348 (523h46m21.168599383s) [upload=459479]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:15:21.588709038 (523h46m21.459792073s) [upload=459479]: Done uploading segments... Dec 27 11:15:21.588743804 (523h46m21.459826839s) [upload=459479]: Upload finished. err: <nil> Duration: 30.003099364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......] duration: 6.002614652s trace: https://jaeger.dev.storj.io/trace/6224b912b66b4383 FAIL (2025-12-27 11:14:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ) context deadline exceeded ======LOGS====== Dec 27 11:08:51.594600280 (523h39m51.465683325s) [upload=459382]: Starting upload Dec 27 11:08:51.594719035 (523h39m51.46580208s) [upload=459382]: Done starting upload Dec 27 11:08:51.594963952 (523h39m51.466046997s) [upload=459382]: Uploading segments... Dec 27 11:08:51.597363922 (523h39m51.468446957s) [upload=459382]: Got next segment. Inline: false Dec 27 11:08:51.597550396 (523h39m51.468633431s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:08:51.597568060 (523h39m51.468651106s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:08:51.845373635 (523h39m51.71645667s) [seg_pos={0 0}, upload=459382]: Begin upload segment... Dec 27 11:08:51.845431365 (523h39m51.7165144s) [seg_pos={0 0}, upload=459382]: Done begin upload segment. Dec 27 11:08:51.845458767 (523h39m51.716541802s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.845505856 (523h39m51.716588901s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846300710 (523h39m51.717383755s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846869614 (523h39m51.717952659s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847171198 (523h39m51.718254233s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847190625 (523h39m51.71827367s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847445951 (523h39m51.718528986s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847733699 (523h39m51.718816734s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848199246 (523h39m51.719282291s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848572185 (523h39m51.71965522s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849138634 (523h39m51.720221669s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849160966 (523h39m51.720244011s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849887350 (523h39m51.720970385s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850333860 (523h39m51.721416895s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850775561 (523h39m51.721858596s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.851324076 (523h39m51.722407111s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:53.470084369 (523h39m53.341167415s) [upload=459382]: Next returned nil segment Dec 27 11:08:53.470123313 (523h39m53.341206349s) [upload=459382]: Waiting for error group managing segments... Dec 27 11:08:53.852166798 (523h39m53.723249843s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.855182762 (523h39m53.726265797s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.859517767 (523h39m53.730600812s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860043207 (523h39m53.731126242s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860290839 (523h39m53.731373884s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860955003 (523h39m53.732038048s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.866529728 (523h39m53.737612773s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.874417208 (523h39m53.745500243s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.877369579 (523h39m53.748452624s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.122780093 (523h39m53.993863138s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.177614417 (523h39m54.048697452s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.234902273 (523h39m54.105985308s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837213205 (523h39m54.70829624s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837234986 (523h39m54.708318031s) [seg_pos={0 0}, upload=459382]: Segment reached optimal threshold of 13 pieces. Dec 27 11:08:54.837538615 (523h39m54.70862166s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection Dec 27 11:08:54.837547552 (523h39m54.708630587s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838051801 (523h39m54.709134846s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838080346 (523h39m54.709163391s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838085315 (523h39m54.70916835s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838093621 (523h39m54.709176656s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838133357 (523h39m54.709216402s) [seg_pos={0 0}, upload=459382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:08:54.838250851 (523h39m54.709333896s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:08:54.838257583 (523h39m54.709340628s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:08:55.134274680 (523h39m55.005357725s) [upload=459382]: Done uploading segments... Dec 27 11:08:55.134300069 (523h39m55.005383124s) [upload=459382]: Upload finished. err: <nil> Duration: 6.002646613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X.....] duration: 5.864620689s trace: https://jaeger.dev.storj.io/trace/6224b912b66b4383 FAIL (2025-12-27 11:13:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ) context deadline exceeded ======LOGS====== Dec 27 11:08:51.594600280 (523h39m51.465683325s) [upload=459382]: Starting upload Dec 27 11:08:51.594719035 (523h39m51.46580208s) [upload=459382]: Done starting upload Dec 27 11:08:51.594963952 (523h39m51.466046997s) [upload=459382]: Uploading segments... Dec 27 11:08:51.597363922 (523h39m51.468446957s) [upload=459382]: Got next segment. Inline: false Dec 27 11:08:51.597550396 (523h39m51.468633431s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:08:51.597568060 (523h39m51.468651106s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:08:51.845373635 (523h39m51.71645667s) [seg_pos={0 0}, upload=459382]: Begin upload segment... Dec 27 11:08:51.845431365 (523h39m51.7165144s) [seg_pos={0 0}, upload=459382]: Done begin upload segment. Dec 27 11:08:51.845458767 (523h39m51.716541802s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.845505856 (523h39m51.716588901s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846300710 (523h39m51.717383755s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846869614 (523h39m51.717952659s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847171198 (523h39m51.718254233s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847190625 (523h39m51.71827367s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847445951 (523h39m51.718528986s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847733699 (523h39m51.718816734s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848199246 (523h39m51.719282291s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848572185 (523h39m51.71965522s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849138634 (523h39m51.720221669s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849160966 (523h39m51.720244011s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849887350 (523h39m51.720970385s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850333860 (523h39m51.721416895s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850775561 (523h39m51.721858596s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.851324076 (523h39m51.722407111s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:53.470084369 (523h39m53.341167415s) [upload=459382]: Next returned nil segment Dec 27 11:08:53.470123313 (523h39m53.341206349s) [upload=459382]: Waiting for error group managing segments... Dec 27 11:08:53.852166798 (523h39m53.723249843s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.855182762 (523h39m53.726265797s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.859517767 (523h39m53.730600812s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860043207 (523h39m53.731126242s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860290839 (523h39m53.731373884s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860955003 (523h39m53.732038048s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.866529728 (523h39m53.737612773s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.874417208 (523h39m53.745500243s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.877369579 (523h39m53.748452624s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.122780093 (523h39m53.993863138s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.177614417 (523h39m54.048697452s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.234902273 (523h39m54.105985308s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837213205 (523h39m54.70829624s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837234986 (523h39m54.708318031s) [seg_pos={0 0}, upload=459382]: Segment reached optimal threshold of 13 pieces. Dec 27 11:08:54.837538615 (523h39m54.70862166s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection Dec 27 11:08:54.837547552 (523h39m54.708630587s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838051801 (523h39m54.709134846s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838080346 (523h39m54.709163391s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838085315 (523h39m54.70916835s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838093621 (523h39m54.709176656s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838133357 (523h39m54.709216402s) [seg_pos={0 0}, upload=459382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:08:54.838250851 (523h39m54.709333896s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:08:54.838257583 (523h39m54.709340628s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:08:55.134274680 (523h39m55.005357725s) [upload=459382]: Done uploading segments... Dec 27 11:08:55.134300069 (523h39m55.005383124s) [upload=459382]: Upload finished. err: <nil> Duration: 5.864648984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X....] duration: 5.673581881s trace: https://jaeger.dev.storj.io/trace/6224b912b66b4383 FAIL (2025-12-27 11:12:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ) context deadline exceeded ======LOGS====== Dec 27 11:08:51.594600280 (523h39m51.465683325s) [upload=459382]: Starting upload Dec 27 11:08:51.594719035 (523h39m51.46580208s) [upload=459382]: Done starting upload Dec 27 11:08:51.594963952 (523h39m51.466046997s) [upload=459382]: Uploading segments... Dec 27 11:08:51.597363922 (523h39m51.468446957s) [upload=459382]: Got next segment. Inline: false Dec 27 11:08:51.597550396 (523h39m51.468633431s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:08:51.597568060 (523h39m51.468651106s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:08:51.845373635 (523h39m51.71645667s) [seg_pos={0 0}, upload=459382]: Begin upload segment... Dec 27 11:08:51.845431365 (523h39m51.7165144s) [seg_pos={0 0}, upload=459382]: Done begin upload segment. Dec 27 11:08:51.845458767 (523h39m51.716541802s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.845505856 (523h39m51.716588901s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846300710 (523h39m51.717383755s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846869614 (523h39m51.717952659s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847171198 (523h39m51.718254233s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847190625 (523h39m51.71827367s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847445951 (523h39m51.718528986s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847733699 (523h39m51.718816734s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848199246 (523h39m51.719282291s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848572185 (523h39m51.71965522s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849138634 (523h39m51.720221669s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849160966 (523h39m51.720244011s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849887350 (523h39m51.720970385s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850333860 (523h39m51.721416895s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850775561 (523h39m51.721858596s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.851324076 (523h39m51.722407111s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:53.470084369 (523h39m53.341167415s) [upload=459382]: Next returned nil segment Dec 27 11:08:53.470123313 (523h39m53.341206349s) [upload=459382]: Waiting for error group managing segments... Dec 27 11:08:53.852166798 (523h39m53.723249843s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.855182762 (523h39m53.726265797s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.859517767 (523h39m53.730600812s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860043207 (523h39m53.731126242s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860290839 (523h39m53.731373884s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860955003 (523h39m53.732038048s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.866529728 (523h39m53.737612773s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.874417208 (523h39m53.745500243s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.877369579 (523h39m53.748452624s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.122780093 (523h39m53.993863138s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.177614417 (523h39m54.048697452s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.234902273 (523h39m54.105985308s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837213205 (523h39m54.70829624s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837234986 (523h39m54.708318031s) [seg_pos={0 0}, upload=459382]: Segment reached optimal threshold of 13 pieces. Dec 27 11:08:54.837538615 (523h39m54.70862166s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection Dec 27 11:08:54.837547552 (523h39m54.708630587s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838051801 (523h39m54.709134846s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838080346 (523h39m54.709163391s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838085315 (523h39m54.70916835s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838093621 (523h39m54.709176656s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838133357 (523h39m54.709216402s) [seg_pos={0 0}, upload=459382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:08:54.838250851 (523h39m54.709333896s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:08:54.838257583 (523h39m54.709340628s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:08:55.134274680 (523h39m55.005357725s) [upload=459382]: Done uploading segments... Dec 27 11:08:55.134300069 (523h39m55.005383124s) [upload=459382]: Upload finished. err: <nil> Duration: 5.673614252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X...] duration: 5.981983247s trace: https://jaeger.dev.storj.io/trace/6224b912b66b4383 FAIL (2025-12-27 11:11:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ) context deadline exceeded ======LOGS====== Dec 27 11:08:51.594600280 (523h39m51.465683325s) [upload=459382]: Starting upload Dec 27 11:08:51.594719035 (523h39m51.46580208s) [upload=459382]: Done starting upload Dec 27 11:08:51.594963952 (523h39m51.466046997s) [upload=459382]: Uploading segments... Dec 27 11:08:51.597363922 (523h39m51.468446957s) [upload=459382]: Got next segment. Inline: false Dec 27 11:08:51.597550396 (523h39m51.468633431s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:08:51.597568060 (523h39m51.468651106s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:08:51.845373635 (523h39m51.71645667s) [seg_pos={0 0}, upload=459382]: Begin upload segment... Dec 27 11:08:51.845431365 (523h39m51.7165144s) [seg_pos={0 0}, upload=459382]: Done begin upload segment. Dec 27 11:08:51.845458767 (523h39m51.716541802s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.845505856 (523h39m51.716588901s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846300710 (523h39m51.717383755s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846869614 (523h39m51.717952659s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847171198 (523h39m51.718254233s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847190625 (523h39m51.71827367s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847445951 (523h39m51.718528986s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847733699 (523h39m51.718816734s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848199246 (523h39m51.719282291s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848572185 (523h39m51.71965522s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849138634 (523h39m51.720221669s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849160966 (523h39m51.720244011s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849887350 (523h39m51.720970385s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850333860 (523h39m51.721416895s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850775561 (523h39m51.721858596s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.851324076 (523h39m51.722407111s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:53.470084369 (523h39m53.341167415s) [upload=459382]: Next returned nil segment Dec 27 11:08:53.470123313 (523h39m53.341206349s) [upload=459382]: Waiting for error group managing segments... Dec 27 11:08:53.852166798 (523h39m53.723249843s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.855182762 (523h39m53.726265797s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.859517767 (523h39m53.730600812s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860043207 (523h39m53.731126242s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860290839 (523h39m53.731373884s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860955003 (523h39m53.732038048s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.866529728 (523h39m53.737612773s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.874417208 (523h39m53.745500243s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.877369579 (523h39m53.748452624s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.122780093 (523h39m53.993863138s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.177614417 (523h39m54.048697452s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.234902273 (523h39m54.105985308s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837213205 (523h39m54.70829624s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837234986 (523h39m54.708318031s) [seg_pos={0 0}, upload=459382]: Segment reached optimal threshold of 13 pieces. Dec 27 11:08:54.837538615 (523h39m54.70862166s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection Dec 27 11:08:54.837547552 (523h39m54.708630587s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838051801 (523h39m54.709134846s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838080346 (523h39m54.709163391s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838085315 (523h39m54.70916835s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838093621 (523h39m54.709176656s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838133357 (523h39m54.709216402s) [seg_pos={0 0}, upload=459382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:08:54.838250851 (523h39m54.709333896s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:08:54.838257583 (523h39m54.709340628s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:08:55.134274680 (523h39m55.005357725s) [upload=459382]: Done uploading segments... Dec 27 11:08:55.134300069 (523h39m55.005383124s) [upload=459382]: Upload finished. err: <nil> Duration: 5.982027781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X..] duration: 18.534457452s trace: https://jaeger.dev.storj.io/trace/6224b912b66b4383 FAIL (2025-12-27 11:11:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ) context deadline exceeded ======LOGS====== Dec 27 11:08:51.594600280 (523h39m51.465683325s) [upload=459382]: Starting upload Dec 27 11:08:51.594719035 (523h39m51.46580208s) [upload=459382]: Done starting upload Dec 27 11:08:51.594963952 (523h39m51.466046997s) [upload=459382]: Uploading segments... Dec 27 11:08:51.597363922 (523h39m51.468446957s) [upload=459382]: Got next segment. Inline: false Dec 27 11:08:51.597550396 (523h39m51.468633431s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:08:51.597568060 (523h39m51.468651106s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:08:51.845373635 (523h39m51.71645667s) [seg_pos={0 0}, upload=459382]: Begin upload segment... Dec 27 11:08:51.845431365 (523h39m51.7165144s) [seg_pos={0 0}, upload=459382]: Done begin upload segment. Dec 27 11:08:51.845458767 (523h39m51.716541802s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.845505856 (523h39m51.716588901s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846300710 (523h39m51.717383755s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846869614 (523h39m51.717952659s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847171198 (523h39m51.718254233s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847190625 (523h39m51.71827367s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847445951 (523h39m51.718528986s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847733699 (523h39m51.718816734s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848199246 (523h39m51.719282291s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848572185 (523h39m51.71965522s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849138634 (523h39m51.720221669s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849160966 (523h39m51.720244011s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849887350 (523h39m51.720970385s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850333860 (523h39m51.721416895s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850775561 (523h39m51.721858596s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.851324076 (523h39m51.722407111s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:53.470084369 (523h39m53.341167415s) [upload=459382]: Next returned nil segment Dec 27 11:08:53.470123313 (523h39m53.341206349s) [upload=459382]: Waiting for error group managing segments... Dec 27 11:08:53.852166798 (523h39m53.723249843s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.855182762 (523h39m53.726265797s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.859517767 (523h39m53.730600812s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860043207 (523h39m53.731126242s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860290839 (523h39m53.731373884s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860955003 (523h39m53.732038048s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.866529728 (523h39m53.737612773s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.874417208 (523h39m53.745500243s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.877369579 (523h39m53.748452624s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.122780093 (523h39m53.993863138s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.177614417 (523h39m54.048697452s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.234902273 (523h39m54.105985308s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837213205 (523h39m54.70829624s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837234986 (523h39m54.708318031s) [seg_pos={0 0}, upload=459382]: Segment reached optimal threshold of 13 pieces. Dec 27 11:08:54.837538615 (523h39m54.70862166s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection Dec 27 11:08:54.837547552 (523h39m54.708630587s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838051801 (523h39m54.709134846s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838080346 (523h39m54.709163391s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838085315 (523h39m54.70916835s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838093621 (523h39m54.709176656s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838133357 (523h39m54.709216402s) [seg_pos={0 0}, upload=459382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:08:54.838250851 (523h39m54.709333896s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:08:54.838257583 (523h39m54.709340628s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:08:55.134274680 (523h39m55.005357725s) [upload=459382]: Done uploading segments... Dec 27 11:08:55.134300069 (523h39m55.005383124s) [upload=459382]: Upload finished. err: <nil> Duration: 18.534494484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X.] duration: 5.978967304s trace: https://jaeger.dev.storj.io/trace/6224b912b66b4383 FAIL (2025-12-27 11:09:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ) context deadline exceeded ======LOGS====== Dec 27 11:08:51.594600280 (523h39m51.465683325s) [upload=459382]: Starting upload Dec 27 11:08:51.594719035 (523h39m51.46580208s) [upload=459382]: Done starting upload Dec 27 11:08:51.594963952 (523h39m51.466046997s) [upload=459382]: Uploading segments... Dec 27 11:08:51.597363922 (523h39m51.468446957s) [upload=459382]: Got next segment. Inline: false Dec 27 11:08:51.597550396 (523h39m51.468633431s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:08:51.597568060 (523h39m51.468651106s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:08:51.845373635 (523h39m51.71645667s) [seg_pos={0 0}, upload=459382]: Begin upload segment... Dec 27 11:08:51.845431365 (523h39m51.7165144s) [seg_pos={0 0}, upload=459382]: Done begin upload segment. Dec 27 11:08:51.845458767 (523h39m51.716541802s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.845505856 (523h39m51.716588901s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846300710 (523h39m51.717383755s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846869614 (523h39m51.717952659s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847171198 (523h39m51.718254233s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847190625 (523h39m51.71827367s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847445951 (523h39m51.718528986s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847733699 (523h39m51.718816734s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848199246 (523h39m51.719282291s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848572185 (523h39m51.71965522s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849138634 (523h39m51.720221669s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849160966 (523h39m51.720244011s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849887350 (523h39m51.720970385s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850333860 (523h39m51.721416895s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850775561 (523h39m51.721858596s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.851324076 (523h39m51.722407111s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:53.470084369 (523h39m53.341167415s) [upload=459382]: Next returned nil segment Dec 27 11:08:53.470123313 (523h39m53.341206349s) [upload=459382]: Waiting for error group managing segments... Dec 27 11:08:53.852166798 (523h39m53.723249843s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.855182762 (523h39m53.726265797s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.859517767 (523h39m53.730600812s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860043207 (523h39m53.731126242s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860290839 (523h39m53.731373884s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860955003 (523h39m53.732038048s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.866529728 (523h39m53.737612773s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.874417208 (523h39m53.745500243s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.877369579 (523h39m53.748452624s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.122780093 (523h39m53.993863138s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.177614417 (523h39m54.048697452s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.234902273 (523h39m54.105985308s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837213205 (523h39m54.70829624s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837234986 (523h39m54.708318031s) [seg_pos={0 0}, upload=459382]: Segment reached optimal threshold of 13 pieces. Dec 27 11:08:54.837538615 (523h39m54.70862166s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection Dec 27 11:08:54.837547552 (523h39m54.708630587s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838051801 (523h39m54.709134846s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838080346 (523h39m54.709163391s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838085315 (523h39m54.70916835s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838093621 (523h39m54.709176656s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838133357 (523h39m54.709216402s) [seg_pos={0 0}, upload=459382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:08:54.838250851 (523h39m54.709333896s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:08:54.838257583 (523h39m54.709340628s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:08:55.134274680 (523h39m55.005357725s) [upload=459382]: Done uploading segments... Dec 27 11:08:55.134300069 (523h39m55.005383124s) [upload=459382]: Upload finished. err: <nil> Duration: 5.978994946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..X] duration: 30.002830452s trace: https://jaeger.dev.storj.io/trace/6224b912b66b4383 FAIL (2025-12-27 11:09:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ) context deadline exceeded ======LOGS====== Dec 27 11:08:51.594600280 (523h39m51.465683325s) [upload=459382]: Starting upload Dec 27 11:08:51.594719035 (523h39m51.46580208s) [upload=459382]: Done starting upload Dec 27 11:08:51.594963952 (523h39m51.466046997s) [upload=459382]: Uploading segments... Dec 27 11:08:51.597363922 (523h39m51.468446957s) [upload=459382]: Got next segment. Inline: false Dec 27 11:08:51.597550396 (523h39m51.468633431s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:08:51.597568060 (523h39m51.468651106s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:08:51.845373635 (523h39m51.71645667s) [seg_pos={0 0}, upload=459382]: Begin upload segment... Dec 27 11:08:51.845431365 (523h39m51.7165144s) [seg_pos={0 0}, upload=459382]: Done begin upload segment. Dec 27 11:08:51.845458767 (523h39m51.716541802s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.845505856 (523h39m51.716588901s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846300710 (523h39m51.717383755s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.846869614 (523h39m51.717952659s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847171198 (523h39m51.718254233s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847190625 (523h39m51.71827367s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847445951 (523h39m51.718528986s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.847733699 (523h39m51.718816734s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848199246 (523h39m51.719282291s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.848572185 (523h39m51.71965522s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849138634 (523h39m51.720221669s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849160966 (523h39m51.720244011s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.849887350 (523h39m51.720970385s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850333860 (523h39m51.721416895s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.850775561 (523h39m51.721858596s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:51.851324076 (523h39m51.722407111s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Uploading piece... Dec 27 11:08:53.470084369 (523h39m53.341167415s) [upload=459382]: Next returned nil segment Dec 27 11:08:53.470123313 (523h39m53.341206349s) [upload=459382]: Waiting for error group managing segments... Dec 27 11:08:53.852166798 (523h39m53.723249843s) [noise=true , address=145.79.227.16:4912 , piece_id=SLI77QSUP36RS3Z5VOXHZ4B3SMPRRDKLU5BQXTCMTFCXEPVM2W4Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.855182762 (523h39m53.726265797s) [noise=true , address=145.79.227.50:4374 , piece_id=7HEDPYCOFEU7B5PXAYR6LJO77YBGT7PTEYBWHMGCKJFXQHM5C4MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.859517767 (523h39m53.730600812s) [noise=true , address=145.79.227.13:4804 , piece_id=RC3ZIV2ZJ3DR7FQFMLIU26VFEWTQSSDFF52FFZHLYZWD4NYFT44Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860043207 (523h39m53.731126242s) [noise=true , address=145.79.227.48:4301 , piece_id=4GXKOVVTVK26UAWO4KFN5T52BN7M3ABBRMC4DEGLXWPQILQBJ5SA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860290839 (523h39m53.731373884s) [noise=true , address=145.79.227.51:4411 , piece_id=LXTLFETUTTWRV6HU5LI3NPN65WK4TTYESO7DLVNROHWUKZEWUCDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.860955003 (523h39m53.732038048s) [noise=true , address=145.79.227.35:4482 , piece_id=25VK5433P53FFRB4KIJCOZGG2EMKIF657E5SJYVZVSFARHZLY67Q, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.866529728 (523h39m53.737612773s) [noise=true , address=145.79.227.46:4601 , piece_id=6VY6D4O3PG3PUSRF6HCJGERJ6YDDRNOP6XMABGDX5XKK3RCVPVFQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.874417208 (523h39m53.745500243s) [noise=true , address=145.79.227.14:4837 , piece_id=KSGPHFLJEEFWCLWJCVGEDP6OSP3F52KGAKADBFYDCNMN2K5NT4VA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:53.877369579 (523h39m53.748452624s) [noise=true , address=145.79.227.7:4295 , piece_id=Q27RQZCPRTRYROBOFGARNICL7LS24UGHQIMYNZFCBEEVB6CPZMYQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.122780093 (523h39m53.993863138s) [noise=true , address=45.91.116.5:4299 , piece_id=FXKCKO7REFNSIZTZQAMEVDNMHXIO3FBBDS6W3O5UHP6QC3Z7T7MQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.177614417 (523h39m54.048697452s) [noise=true , address=45.91.116.12:4228 , piece_id=5TLL7ZRK3UCHC3UKUDDMBZD2HG5I2WUEMZJZ4DSMOFGZVY4FULLA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.234902273 (523h39m54.105985308s) [noise=true , address=45.91.116.6:4976 , piece_id=HYHOLYVUK2IEIRVLF37OYOQABJOBD4CU4PLTUODC7JHSRLCYQKIA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837213205 (523h39m54.70829624s) [noise=true , address=45.91.116.14:4192 , piece_id=3LKGOQOY4DPVTU7QQJ3PSVPIQXF3QGB76KEQS5X44VTUYM6LEQDQ, seg_pos={0 0}, upload=459382]: Done uploading piece. err: <nil> Dec 27 11:08:54.837234986 (523h39m54.708318031s) [seg_pos={0 0}, upload=459382]: Segment reached optimal threshold of 13 pieces. Dec 27 11:08:54.837538615 (523h39m54.70862166s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59012->77.90.31.15:4700: use of closed network connection Dec 27 11:08:54.837547552 (523h39m54.708630587s) [noise=true , address=77.90.31.15:4700 , piece_id=G2WQVOFWVT6XGYFTT63PBJTV6Q4KDQM2HDRQSGEFBQOVXOAZVDFA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838051801 (523h39m54.709134846s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838080346 (523h39m54.709163391s) [noise=true , address=145.79.227.44:4194 , piece_id=KH6G4DRZAQ3JIARPX7MXDPWWO5MPLEWHCDPRXSO6JXDBCTZ63ZJA, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838085315 (523h39m54.70916835s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 11:08:54.838093621 (523h39m54.709176656s) [noise=true , address=45.91.116.8:4047 , piece_id=N4PEXAIPQWRK2WHUEUM3DYUM372KEP57YM6M2UJJI4LYXE5FRG2A, seg_pos={0 0}, upload=459382]: Long tail cancellation due to optimal threshold Dec 27 11:08:54.838133357 (523h39m54.709216402s) [seg_pos={0 0}, upload=459382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:08:54.838250851 (523h39m54.709333896s) [upload=459382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:08:54.838257583 (523h39m54.709340628s) [upload=459382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:08:55.134274680 (523h39m55.005357725s) [upload=459382]: Done uploading segments... Dec 27 11:08:55.134300069 (523h39m55.005383124s) [upload=459382]: Upload finished. err: <nil> Duration: 30.002932327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX..] duration: 5.607129452s trace: https://jaeger.dev.storj.io/trace/3f347a6322556e11 FAIL (2025-12-27 11:07:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ) context deadline exceeded ======LOGS====== Dec 27 11:05:56.391771716 (523h36m56.262854762s) [upload=459340]: Starting upload Dec 27 11:05:56.391849294 (523h36m56.262932339s) [upload=459340]: Done starting upload Dec 27 11:05:56.391990763 (523h36m56.263073818s) [upload=459340]: Uploading segments... Dec 27 11:05:56.392099370 (523h36m56.263182435s) [upload=459340]: Got next segment. Inline: false Dec 27 11:05:56.392163763 (523h36m56.263246808s) [upload=459340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:05:56.392171237 (523h36m56.263254282s) [upload=459340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:05:56.637632267 (523h36m56.508715302s) [seg_pos={0 0}, upload=459340]: Begin upload segment... Dec 27 11:05:56.637677342 (523h36m56.508760387s) [seg_pos={0 0}, upload=459340]: Done begin upload segment. Dec 27 11:05:56.637700797 (523h36m56.508783842s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.637828430 (523h36m56.508911475s) [noise=true , address=145.79.227.15:4874 , piece_id=5RBUCGSLS3D3RFD7A6YWJTBA3UKSRHSFHWQY4IF2E7QF5RXL2W7Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.638602073 (523h36m56.509685108s) [noise=true , address=45.91.116.5:4297 , piece_id=WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639040347 (523h36m56.510123382s) [noise=true , address=145.79.227.25:4617 , piece_id=J2YGWRNP5YDKEOK44IFJLVXOEA4V35GXDUUIU5ZSS77F5TCVXNAQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639464836 (523h36m56.510547871s) [noise=true , address=145.79.227.37:4567 , piece_id=4HXFO54IVI57GNWLFNNVSGDMQOYBQZNQ4ESLTG2WNAVHZMPAOGYA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639900255 (523h36m56.51098329s) [noise=true , address=77.90.31.14:4664 , piece_id=DL6PBEHU5HCP6W2ISDLHYNYQ2B3LAOGCJJZZKAT4XXZIOQBD4UOQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.640356895 (523h36m56.51143994s) [noise=true , address=145.79.227.8:4331 , piece_id=HOZH445CKWPF7DQ2PIBRDZZCUEB6VII46SYKR435UD45PSM6F6EA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.641353713 (523h36m56.512436748s) [noise=true , address=77.90.31.10:4520 , piece_id=GPVNLIIOHKKZYFN6ZDSKIMXE4XRDBAXJVEABGFHFQKA4VOKMXISQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.642014351 (523h36m56.513097386s) [noise=true , address=77.90.31.13:4627 , piece_id=XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.642759019 (523h36m56.513842054s) [noise=true , address=145.79.227.43:4159 , piece_id=KHVMA3RZYW4E34AGFQ2CEVYKMM3XQ4FUBLWYA5RTDFJGHI2TCR2Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.643821361 (523h36m56.514904396s) [noise=true , address=145.79.227.3:4150 , piece_id=UKQYNJX6MXCDEGZL4PKCWTB6JDRKRZHYZJEPDH4LJERE3GOPEPVQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.644876591 (523h36m56.515959636s) [noise=true , address=145.79.227.23:4546 , piece_id=DMBLZGZKMGTAPEIT5LBQZFTHHV4I6FM5AN2AJWSJ7ARNDA33P62A, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.645997976 (523h36m56.517081021s) [noise=true , address=145.79.227.47:4265 , piece_id=KODLBPOA2NVV32IN6TWPXOABE2WEYIRU3UMRK42EHV4G45QHXUFQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.646432964 (523h36m56.517515999s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.649142974 (523h36m56.520226029s) [noise=true , address=145.79.227.2:4113 , piece_id=LYQL32EB5SKYSPZXIQYPFJR6MKTREGYHHY2WNEY32JTHUO2RS3XQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.654581060 (523h36m56.525664105s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:58.257446276 (523h36m58.128529321s) [upload=459340]: Next returned nil segment Dec 27 11:05:58.257459741 (523h36m58.128542786s) [upload=459340]: Waiting for error group managing segments... Dec 27 11:05:58.649958766 (523h36m58.521041811s) [noise=true , address=145.79.227.15:4874 , piece_id=5RBUCGSLS3D3RFD7A6YWJTBA3UKSRHSFHWQY4IF2E7QF5RXL2W7Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.650039940 (523h36m58.521122975s) [noise=true , address=145.79.227.3:4150 , piece_id=UKQYNJX6MXCDEGZL4PKCWTB6JDRKRZHYZJEPDH4LJERE3GOPEPVQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.653221990 (523h36m58.524305035s) [noise=true , address=145.79.227.37:4567 , piece_id=4HXFO54IVI57GNWLFNNVSGDMQOYBQZNQ4ESLTG2WNAVHZMPAOGYA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.654675166 (523h36m58.525758211s) [noise=true , address=145.79.227.25:4617 , piece_id=J2YGWRNP5YDKEOK44IFJLVXOEA4V35GXDUUIU5ZSS77F5TCVXNAQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.658547401 (523h36m58.529630436s) [noise=true , address=145.79.227.23:4546 , piece_id=DMBLZGZKMGTAPEIT5LBQZFTHHV4I6FM5AN2AJWSJ7ARNDA33P62A, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.661603259 (523h36m58.532686294s) [noise=true , address=145.79.227.2:4113 , piece_id=LYQL32EB5SKYSPZXIQYPFJR6MKTREGYHHY2WNEY32JTHUO2RS3XQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.664221906 (523h36m58.535304951s) [noise=true , address=145.79.227.8:4331 , piece_id=HOZH445CKWPF7DQ2PIBRDZZCUEB6VII46SYKR435UD45PSM6F6EA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.664542867 (523h36m58.535625902s) [noise=true , address=145.79.227.43:4159 , piece_id=KHVMA3RZYW4E34AGFQ2CEVYKMM3XQ4FUBLWYA5RTDFJGHI2TCR2Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.669995710 (523h36m58.541078755s) [noise=true , address=145.79.227.47:4265 , piece_id=KODLBPOA2NVV32IN6TWPXOABE2WEYIRU3UMRK42EHV4G45QHXUFQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.853120901 (523h36m58.724203946s) [noise=true , address=77.90.31.10:4520 , piece_id=GPVNLIIOHKKZYFN6ZDSKIMXE4XRDBAXJVEABGFHFQKA4VOKMXISQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.896784864 (523h36m58.767867899s) [noise=true , address=77.90.31.14:4664 , piece_id=DL6PBEHU5HCP6W2ISDLHYNYQ2B3LAOGCJJZZKAT4XXZIOQBD4UOQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.919957709 (523h36m58.791040755s) [noise=true , address=77.90.31.13:4627 , piece_id=XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:59.719594795 (523h36m59.59067783s) [noise=true , address=45.91.116.5:4297 , piece_id=WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:59.719615103 (523h36m59.590698148s) [seg_pos={0 0}, upload=459340]: Segment reached optimal threshold of 13 pieces. Dec 27 11:05:59.719831666 (523h36m59.590914701s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.719840172 (523h36m59.590923207s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.719966643 (523h36m59.591049688s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.719979206 (523h36m59.591062252s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.720058978 (523h36m59.591142013s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.720066823 (523h36m59.591149868s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.720114184 (523h36m59.591197219s) [seg_pos={0 0}, upload=459340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:05:59.720213252 (523h36m59.591296287s) [upload=459340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:05:59.720221358 (523h36m59.591304403s) [upload=459340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:06:00.015589799 (523h36m59.886672854s) [upload=459340]: Done uploading segments... Dec 27 11:06:00.015613393 (523h36m59.886696438s) [upload=459340]: Upload finished. err: <nil> Duration: 5.607158246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX.] duration: 5.834699753s trace: https://jaeger.dev.storj.io/trace/3f347a6322556e11 FAIL (2025-12-27 11:07:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ) context deadline exceeded ======LOGS====== Dec 27 11:05:56.391771716 (523h36m56.262854762s) [upload=459340]: Starting upload Dec 27 11:05:56.391849294 (523h36m56.262932339s) [upload=459340]: Done starting upload Dec 27 11:05:56.391990763 (523h36m56.263073818s) [upload=459340]: Uploading segments... Dec 27 11:05:56.392099370 (523h36m56.263182435s) [upload=459340]: Got next segment. Inline: false Dec 27 11:05:56.392163763 (523h36m56.263246808s) [upload=459340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:05:56.392171237 (523h36m56.263254282s) [upload=459340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:05:56.637632267 (523h36m56.508715302s) [seg_pos={0 0}, upload=459340]: Begin upload segment... Dec 27 11:05:56.637677342 (523h36m56.508760387s) [seg_pos={0 0}, upload=459340]: Done begin upload segment. Dec 27 11:05:56.637700797 (523h36m56.508783842s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.637828430 (523h36m56.508911475s) [noise=true , address=145.79.227.15:4874 , piece_id=5RBUCGSLS3D3RFD7A6YWJTBA3UKSRHSFHWQY4IF2E7QF5RXL2W7Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.638602073 (523h36m56.509685108s) [noise=true , address=45.91.116.5:4297 , piece_id=WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639040347 (523h36m56.510123382s) [noise=true , address=145.79.227.25:4617 , piece_id=J2YGWRNP5YDKEOK44IFJLVXOEA4V35GXDUUIU5ZSS77F5TCVXNAQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639464836 (523h36m56.510547871s) [noise=true , address=145.79.227.37:4567 , piece_id=4HXFO54IVI57GNWLFNNVSGDMQOYBQZNQ4ESLTG2WNAVHZMPAOGYA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639900255 (523h36m56.51098329s) [noise=true , address=77.90.31.14:4664 , piece_id=DL6PBEHU5HCP6W2ISDLHYNYQ2B3LAOGCJJZZKAT4XXZIOQBD4UOQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.640356895 (523h36m56.51143994s) [noise=true , address=145.79.227.8:4331 , piece_id=HOZH445CKWPF7DQ2PIBRDZZCUEB6VII46SYKR435UD45PSM6F6EA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.641353713 (523h36m56.512436748s) [noise=true , address=77.90.31.10:4520 , piece_id=GPVNLIIOHKKZYFN6ZDSKIMXE4XRDBAXJVEABGFHFQKA4VOKMXISQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.642014351 (523h36m56.513097386s) [noise=true , address=77.90.31.13:4627 , piece_id=XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.642759019 (523h36m56.513842054s) [noise=true , address=145.79.227.43:4159 , piece_id=KHVMA3RZYW4E34AGFQ2CEVYKMM3XQ4FUBLWYA5RTDFJGHI2TCR2Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.643821361 (523h36m56.514904396s) [noise=true , address=145.79.227.3:4150 , piece_id=UKQYNJX6MXCDEGZL4PKCWTB6JDRKRZHYZJEPDH4LJERE3GOPEPVQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.644876591 (523h36m56.515959636s) [noise=true , address=145.79.227.23:4546 , piece_id=DMBLZGZKMGTAPEIT5LBQZFTHHV4I6FM5AN2AJWSJ7ARNDA33P62A, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.645997976 (523h36m56.517081021s) [noise=true , address=145.79.227.47:4265 , piece_id=KODLBPOA2NVV32IN6TWPXOABE2WEYIRU3UMRK42EHV4G45QHXUFQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.646432964 (523h36m56.517515999s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.649142974 (523h36m56.520226029s) [noise=true , address=145.79.227.2:4113 , piece_id=LYQL32EB5SKYSPZXIQYPFJR6MKTREGYHHY2WNEY32JTHUO2RS3XQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.654581060 (523h36m56.525664105s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:58.257446276 (523h36m58.128529321s) [upload=459340]: Next returned nil segment Dec 27 11:05:58.257459741 (523h36m58.128542786s) [upload=459340]: Waiting for error group managing segments... Dec 27 11:05:58.649958766 (523h36m58.521041811s) [noise=true , address=145.79.227.15:4874 , piece_id=5RBUCGSLS3D3RFD7A6YWJTBA3UKSRHSFHWQY4IF2E7QF5RXL2W7Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.650039940 (523h36m58.521122975s) [noise=true , address=145.79.227.3:4150 , piece_id=UKQYNJX6MXCDEGZL4PKCWTB6JDRKRZHYZJEPDH4LJERE3GOPEPVQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.653221990 (523h36m58.524305035s) [noise=true , address=145.79.227.37:4567 , piece_id=4HXFO54IVI57GNWLFNNVSGDMQOYBQZNQ4ESLTG2WNAVHZMPAOGYA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.654675166 (523h36m58.525758211s) [noise=true , address=145.79.227.25:4617 , piece_id=J2YGWRNP5YDKEOK44IFJLVXOEA4V35GXDUUIU5ZSS77F5TCVXNAQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.658547401 (523h36m58.529630436s) [noise=true , address=145.79.227.23:4546 , piece_id=DMBLZGZKMGTAPEIT5LBQZFTHHV4I6FM5AN2AJWSJ7ARNDA33P62A, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.661603259 (523h36m58.532686294s) [noise=true , address=145.79.227.2:4113 , piece_id=LYQL32EB5SKYSPZXIQYPFJR6MKTREGYHHY2WNEY32JTHUO2RS3XQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.664221906 (523h36m58.535304951s) [noise=true , address=145.79.227.8:4331 , piece_id=HOZH445CKWPF7DQ2PIBRDZZCUEB6VII46SYKR435UD45PSM6F6EA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.664542867 (523h36m58.535625902s) [noise=true , address=145.79.227.43:4159 , piece_id=KHVMA3RZYW4E34AGFQ2CEVYKMM3XQ4FUBLWYA5RTDFJGHI2TCR2Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.669995710 (523h36m58.541078755s) [noise=true , address=145.79.227.47:4265 , piece_id=KODLBPOA2NVV32IN6TWPXOABE2WEYIRU3UMRK42EHV4G45QHXUFQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.853120901 (523h36m58.724203946s) [noise=true , address=77.90.31.10:4520 , piece_id=GPVNLIIOHKKZYFN6ZDSKIMXE4XRDBAXJVEABGFHFQKA4VOKMXISQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.896784864 (523h36m58.767867899s) [noise=true , address=77.90.31.14:4664 , piece_id=DL6PBEHU5HCP6W2ISDLHYNYQ2B3LAOGCJJZZKAT4XXZIOQBD4UOQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.919957709 (523h36m58.791040755s) [noise=true , address=77.90.31.13:4627 , piece_id=XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:59.719594795 (523h36m59.59067783s) [noise=true , address=45.91.116.5:4297 , piece_id=WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:59.719615103 (523h36m59.590698148s) [seg_pos={0 0}, upload=459340]: Segment reached optimal threshold of 13 pieces. Dec 27 11:05:59.719831666 (523h36m59.590914701s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.719840172 (523h36m59.590923207s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.719966643 (523h36m59.591049688s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.719979206 (523h36m59.591062252s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.720058978 (523h36m59.591142013s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.720066823 (523h36m59.591149868s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.720114184 (523h36m59.591197219s) [seg_pos={0 0}, upload=459340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:05:59.720213252 (523h36m59.591296287s) [upload=459340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:05:59.720221358 (523h36m59.591304403s) [upload=459340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:06:00.015589799 (523h36m59.886672854s) [upload=459340]: Done uploading segments... Dec 27 11:06:00.015613393 (523h36m59.886696438s) [upload=459340]: Upload finished. err: <nil> Duration: 5.834725883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..XX] duration: 30.002962775s trace: https://jaeger.dev.storj.io/trace/3f347a6322556e11 FAIL (2025-12-27 11:06:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ) context deadline exceeded ======LOGS====== Dec 27 11:05:56.391771716 (523h36m56.262854762s) [upload=459340]: Starting upload Dec 27 11:05:56.391849294 (523h36m56.262932339s) [upload=459340]: Done starting upload Dec 27 11:05:56.391990763 (523h36m56.263073818s) [upload=459340]: Uploading segments... Dec 27 11:05:56.392099370 (523h36m56.263182435s) [upload=459340]: Got next segment. Inline: false Dec 27 11:05:56.392163763 (523h36m56.263246808s) [upload=459340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:05:56.392171237 (523h36m56.263254282s) [upload=459340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:05:56.637632267 (523h36m56.508715302s) [seg_pos={0 0}, upload=459340]: Begin upload segment... Dec 27 11:05:56.637677342 (523h36m56.508760387s) [seg_pos={0 0}, upload=459340]: Done begin upload segment. Dec 27 11:05:56.637700797 (523h36m56.508783842s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.637828430 (523h36m56.508911475s) [noise=true , address=145.79.227.15:4874 , piece_id=5RBUCGSLS3D3RFD7A6YWJTBA3UKSRHSFHWQY4IF2E7QF5RXL2W7Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.638602073 (523h36m56.509685108s) [noise=true , address=45.91.116.5:4297 , piece_id=WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639040347 (523h36m56.510123382s) [noise=true , address=145.79.227.25:4617 , piece_id=J2YGWRNP5YDKEOK44IFJLVXOEA4V35GXDUUIU5ZSS77F5TCVXNAQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639464836 (523h36m56.510547871s) [noise=true , address=145.79.227.37:4567 , piece_id=4HXFO54IVI57GNWLFNNVSGDMQOYBQZNQ4ESLTG2WNAVHZMPAOGYA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.639900255 (523h36m56.51098329s) [noise=true , address=77.90.31.14:4664 , piece_id=DL6PBEHU5HCP6W2ISDLHYNYQ2B3LAOGCJJZZKAT4XXZIOQBD4UOQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.640356895 (523h36m56.51143994s) [noise=true , address=145.79.227.8:4331 , piece_id=HOZH445CKWPF7DQ2PIBRDZZCUEB6VII46SYKR435UD45PSM6F6EA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.641353713 (523h36m56.512436748s) [noise=true , address=77.90.31.10:4520 , piece_id=GPVNLIIOHKKZYFN6ZDSKIMXE4XRDBAXJVEABGFHFQKA4VOKMXISQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.642014351 (523h36m56.513097386s) [noise=true , address=77.90.31.13:4627 , piece_id=XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.642759019 (523h36m56.513842054s) [noise=true , address=145.79.227.43:4159 , piece_id=KHVMA3RZYW4E34AGFQ2CEVYKMM3XQ4FUBLWYA5RTDFJGHI2TCR2Q, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.643821361 (523h36m56.514904396s) [noise=true , address=145.79.227.3:4150 , piece_id=UKQYNJX6MXCDEGZL4PKCWTB6JDRKRZHYZJEPDH4LJERE3GOPEPVQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.644876591 (523h36m56.515959636s) [noise=true , address=145.79.227.23:4546 , piece_id=DMBLZGZKMGTAPEIT5LBQZFTHHV4I6FM5AN2AJWSJ7ARNDA33P62A, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.645997976 (523h36m56.517081021s) [noise=true , address=145.79.227.47:4265 , piece_id=KODLBPOA2NVV32IN6TWPXOABE2WEYIRU3UMRK42EHV4G45QHXUFQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.646432964 (523h36m56.517515999s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.649142974 (523h36m56.520226029s) [noise=true , address=145.79.227.2:4113 , piece_id=LYQL32EB5SKYSPZXIQYPFJR6MKTREGYHHY2WNEY32JTHUO2RS3XQ, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:56.654581060 (523h36m56.525664105s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Uploading piece... Dec 27 11:05:58.257446276 (523h36m58.128529321s) [upload=459340]: Next returned nil segment Dec 27 11:05:58.257459741 (523h36m58.128542786s) [upload=459340]: Waiting for error group managing segments... Dec 27 11:05:58.649958766 (523h36m58.521041811s) [noise=true , address=145.79.227.15:4874 , piece_id=5RBUCGSLS3D3RFD7A6YWJTBA3UKSRHSFHWQY4IF2E7QF5RXL2W7Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.650039940 (523h36m58.521122975s) [noise=true , address=145.79.227.3:4150 , piece_id=UKQYNJX6MXCDEGZL4PKCWTB6JDRKRZHYZJEPDH4LJERE3GOPEPVQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.653221990 (523h36m58.524305035s) [noise=true , address=145.79.227.37:4567 , piece_id=4HXFO54IVI57GNWLFNNVSGDMQOYBQZNQ4ESLTG2WNAVHZMPAOGYA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.654675166 (523h36m58.525758211s) [noise=true , address=145.79.227.25:4617 , piece_id=J2YGWRNP5YDKEOK44IFJLVXOEA4V35GXDUUIU5ZSS77F5TCVXNAQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.658547401 (523h36m58.529630436s) [noise=true , address=145.79.227.23:4546 , piece_id=DMBLZGZKMGTAPEIT5LBQZFTHHV4I6FM5AN2AJWSJ7ARNDA33P62A, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.661603259 (523h36m58.532686294s) [noise=true , address=145.79.227.2:4113 , piece_id=LYQL32EB5SKYSPZXIQYPFJR6MKTREGYHHY2WNEY32JTHUO2RS3XQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.664221906 (523h36m58.535304951s) [noise=true , address=145.79.227.8:4331 , piece_id=HOZH445CKWPF7DQ2PIBRDZZCUEB6VII46SYKR435UD45PSM6F6EA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.664542867 (523h36m58.535625902s) [noise=true , address=145.79.227.43:4159 , piece_id=KHVMA3RZYW4E34AGFQ2CEVYKMM3XQ4FUBLWYA5RTDFJGHI2TCR2Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.669995710 (523h36m58.541078755s) [noise=true , address=145.79.227.47:4265 , piece_id=KODLBPOA2NVV32IN6TWPXOABE2WEYIRU3UMRK42EHV4G45QHXUFQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.853120901 (523h36m58.724203946s) [noise=true , address=77.90.31.10:4520 , piece_id=GPVNLIIOHKKZYFN6ZDSKIMXE4XRDBAXJVEABGFHFQKA4VOKMXISQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.896784864 (523h36m58.767867899s) [noise=true , address=77.90.31.14:4664 , piece_id=DL6PBEHU5HCP6W2ISDLHYNYQ2B3LAOGCJJZZKAT4XXZIOQBD4UOQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:58.919957709 (523h36m58.791040755s) [noise=true , address=77.90.31.13:4627 , piece_id=XVJGG3YNNC7AIHHOYB77FSZBQJYYT7GA6IJBVK5ELTYIJX2ZNGKQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:59.719594795 (523h36m59.59067783s) [noise=true , address=45.91.116.5:4297 , piece_id=WTR4BGAWJKYRHJ6NMTGHXCYUIQSJOTG3WUXIJOIHOOV7SJJWTE4A, seg_pos={0 0}, upload=459340]: Done uploading piece. err: <nil> Dec 27 11:05:59.719615103 (523h36m59.590698148s) [seg_pos={0 0}, upload=459340]: Segment reached optimal threshold of 13 pieces. Dec 27 11:05:59.719831666 (523h36m59.590914701s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.719840172 (523h36m59.590923207s) [noise=true , address=145.79.227.30:4761 , piece_id=XCIBUOCMB7C4LSIXUXNVYRVCAH3YTEC6P62NNURFMOOX72TSHPGA, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.719966643 (523h36m59.591049688s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.719979206 (523h36m59.591062252s) [noise=true , address=45.91.116.6:4975 , piece_id=SPMVT3VRUXDLSEF2WCBHRCIDZO3ZBE5YAK77SBO6P5YDMTDNBB4Q, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.720058978 (523h36m59.591142013s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 11:05:59.720066823 (523h36m59.591149868s) [noise=true , address=145.79.227.34:4907 , piece_id=CRSDLD5DKUFMQBTQLLIUE4HOMJM3B3S2YIDCPTK6IFXSWM2Y5KRQ, seg_pos={0 0}, upload=459340]: Long tail cancellation due to optimal threshold Dec 27 11:05:59.720114184 (523h36m59.591197219s) [seg_pos={0 0}, upload=459340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:05:59.720213252 (523h36m59.591296287s) [upload=459340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:05:59.720221358 (523h36m59.591304403s) [upload=459340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:06:00.015589799 (523h36m59.886672854s) [upload=459340]: Done uploading segments... Dec 27 11:06:00.015613393 (523h36m59.886696438s) [upload=459340]: Upload finished. err: <nil> Duration: 30.002986731s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..X] duration: 30.002217725s trace: https://jaeger.dev.storj.io/trace/14fa7633cd4cd850 FAIL (2025-12-27 11:05:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: WYD6XN5GJKWQMS7A3L5HGLPKDZXWBNW4ZH2YSBTGCAZS7PEXSO2Q) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 6IIFEL26BVWNPUDVCBZVS3IYCP2BDTXZVWRTQMS6RDAOSCTZYPFA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: W2UYHGQ7WNAGFGGFC2T64RX7PMDVWWNQXRALQZKKI7FNT7URQYSQ) context deadline exceeded ======LOGS====== 27 11:04:46.393982916 (523h35m46.265065951s) [upload=459326]: Starting upload Dec 27 11:04:46.394070704 (523h35m46.265153759s) [upload=459326]: Done starting upload Dec 27 11:04:46.394140225 (523h35m46.265223271s) [upload=459326]: Uploading segments... Dec 27 11:04:46.394536040 (523h35m46.265619095s) [upload=459326]: Got next segment. Inline: false Dec 27 11:04:46.394575124 (523h35m46.265658159s) [upload=459326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:04:46.394580784 (523h35m46.26566382s) [upload=459326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:04:46.641737873 (523h35m46.512820918s) [seg_pos={0 0}, upload=459326]: Begin upload segment... Dec 27 11:04:46.641916944 (523h35m46.512999979s) [seg_pos={0 0}, upload=459326]: Done begin upload segment. Dec 27 11:04:46.641963584 (523h35m46.513046629s) [noise=true , address=45.91.116.6:4973 , piece_id=FJNZX5JAXCYYO6EQXDVI5AN3EMB365FSZGDAQKWOKQMMVBBNQSVA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.642025330 (523h35m46.513108375s) [noise=true , address=145.79.227.13:4802 , piece_id=A2YKMBFLXNMBR6J75FKKJJDCTO3PMZ3I7XAPDACBKCTI7Y3HQPMQ, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.642602279 (523h35m46.513685314s) [noise=true , address=77.90.31.17:4770 , piece_id=W2UYHGQ7WNAGFGGFC2T64RX7PMDVWWNQXRALQZKKI7FNT7URQYSQ, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.643133410 (523h35m46.514216446s) [noise=true , address=145.79.227.7:4295 , piece_id=G3UMG72FBKRNHD23WDVMF2SFFQJQLZ2U2EORUH755UXONNLVXEPQ, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.643543021 (523h35m46.514626056s) [noise=true , address=145.79.227.4:4185 , piece_id=2GUTHQJRLL44RM6KEKWMBOINJ4FQIU44OGN2DJ5MC5V26GXXC6MQ, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.643969052 (523h35m46.515052087s) [noise=true , address=45.91.116.9:4261 , piece_id=G62JHNAEU57DI56EXAI4QDSVABPJILRCD5DVKY3GKONGS2LWCRRA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.644549798 (523h35m46.515632843s) [noise=true , address=145.79.227.48:4301 , piece_id=742GQZRFFELX4FOVWUUXNUHZPC4EMBYM23QYVLVHZ453FC537RFA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.645084616 (523h35m46.516167651s) [noise=true , address=45.91.116.11:4120 , piece_id=WYD6XN5GJKWQMS7A3L5HGLPKDZXWBNW4ZH2YSBTGCAZS7PEXSO2Q, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.646319237 (523h35m46.517402282s) [noise=true , address=77.90.31.21:4877 , piece_id=NWU2EVA22Z67KKBTFA5CDUDRRMHY3A32NVU7WR5VNUZADTRWY26Q, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.646634277 (523h35m46.517717312s) [noise=true , address=145.79.227.47:4265 , piece_id=4D6OM4J3FTWQ37DYGMQ3J52LXUNJFILHHKOJE5RYDR2YUWZJAISA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.647845554 (523h35m46.518928589s) [noise=true , address=77.90.31.18:4808 , piece_id=NDXVE5Q24QDWDIGIUUP3UAKUXJGVBBQQF3NDTBZY3QP7ONURKUSA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.651279031 (523h35m46.522362076s) [noise=true , address=77.90.31.19:4842 , piece_id=XWXLG22SJWP7SCCKKQURMMVV5J47IA2IXRT55YNVYWAFTMCK6DDA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.655143109 (523h35m46.526226154s) [noise=true , address=145.79.227.39:4014 , piece_id=GYTWUGU4CZBPE3VZTLBCDWDEKE64K4TQ6OSX4PGDLOP4WOWAVRFA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.657741547 (523h35m46.528824582s) [noise=true , address=77.90.31.16:4735 , piece_id=6IIFEL26BVWNPUDVCBZVS3IYCP2BDTXZVWRTQMS6RDAOSCTZYPFA, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.661639209 (523h35m46.532722254s) [noise=true , address=145.79.227.16:4911 , piece_id=TLPZI7B7TQHX4ER7ZUJ4A6QFP7Q6AKHQY5JFO3JECIVWZE5FEORQ, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:46.662024292 (523h35m46.533107327s) [noise=true , address=145.79.227.2:4113 , piece_id=CRYY33KOCI5XB5SEGYTBSEO2355WUEFLQD5G2N7NXNAPAZKCA5MQ, seg_pos={0 0}, upload=459326]: Uploading piece... Dec 27 11:04:48.370235485 (523h35m48.24131853s) [upload=459326]: Next returned nil segment Dec 27 11:04:48.370249252 (523h35m48.241332297s) [upload=459326]: Waiting for error group managing segments... Dec 27 11:04:48.654614421 (523h35m48.525697456s) [noise=true , address=145.79.227.7:4295 , piece_id=G3UMG72FBKRNHD23WDVMF2SFFQJQLZ2U2EORUH755UXONNLVXEPQ, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.666092687 (523h35m48.537175732s) [noise=true , address=145.79.227.39:4014 , piece_id=GYTWUGU4CZBPE3VZTLBCDWDEKE64K4TQ6OSX4PGDLOP4WOWAVRFA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.671176187 (523h35m48.542259222s) [noise=true , address=145.79.227.4:4185 , piece_id=2GUTHQJRLL44RM6KEKWMBOINJ4FQIU44OGN2DJ5MC5V26GXXC6MQ, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.671270918 (523h35m48.542353963s) [noise=true , address=145.79.227.13:4802 , piece_id=A2YKMBFLXNMBR6J75FKKJJDCTO3PMZ3I7XAPDACBKCTI7Y3HQPMQ, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.672504446 (523h35m48.543587491s) [noise=true , address=145.79.227.48:4301 , piece_id=742GQZRFFELX4FOVWUUXNUHZPC4EMBYM23QYVLVHZ453FC537RFA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.677104906 (523h35m48.548187941s) [noise=true , address=145.79.227.2:4113 , piece_id=CRYY33KOCI5XB5SEGYTBSEO2355WUEFLQD5G2N7NXNAPAZKCA5MQ, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.678514420 (523h35m48.549597455s) [noise=true , address=145.79.227.16:4911 , piece_id=TLPZI7B7TQHX4ER7ZUJ4A6QFP7Q6AKHQY5JFO3JECIVWZE5FEORQ, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.693354278 (523h35m48.564437313s) [noise=true , address=145.79.227.47:4265 , piece_id=4D6OM4J3FTWQ37DYGMQ3J52LXUNJFILHHKOJE5RYDR2YUWZJAISA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.881229845 (523h35m48.75231288s) [noise=true , address=77.90.31.16:4735 , piece_id=6IIFEL26BVWNPUDVCBZVS3IYCP2BDTXZVWRTQMS6RDAOSCTZYPFA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:48.931011337 (523h35m48.802094372s) [noise=true , address=45.91.116.11:4120 , piece_id=WYD6XN5GJKWQMS7A3L5HGLPKDZXWBNW4ZH2YSBTGCAZS7PEXSO2Q, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:49.015748380 (523h35m48.886831426s) [noise=true , address=77.90.31.17:4770 , piece_id=W2UYHGQ7WNAGFGGFC2T64RX7PMDVWWNQXRALQZKKI7FNT7URQYSQ, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:49.020535216 (523h35m48.891618261s) [noise=true , address=77.90.31.21:4877 , piece_id=NWU2EVA22Z67KKBTFA5CDUDRRMHY3A32NVU7WR5VNUZADTRWY26Q, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:49.172710253 (523h35m49.043793288s) [noise=true , address=45.91.116.9:4261 , piece_id=G62JHNAEU57DI56EXAI4QDSVABPJILRCD5DVKY3GKONGS2LWCRRA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: <nil> Dec 27 11:04:49.172732446 (523h35m49.043815481s) [seg_pos={0 0}, upload=459326]: Segment reached optimal threshold of 13 pieces. Dec 27 11:04:49.172896448 (523h35m49.043979483s) [noise=true , address=77.90.31.18:4808 , piece_id=NDXVE5Q24QDWDIGIUUP3UAKUXJGVBBQQF3NDTBZY3QP7ONURKUSA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 11:04:49.172903161 (523h35m49.043986206s) [noise=true , address=77.90.31.18:4808 , piece_id=NDXVE5Q24QDWDIGIUUP3UAKUXJGVBBQQF3NDTBZY3QP7ONURKUSA, seg_pos={0 0}, upload=459326]: Long tail cancellation due to optimal threshold Dec 27 11:04:49.173122328 (523h35m49.044205363s) [noise=true , address=77.90.31.19:4842 , piece_id=XWXLG22SJWP7SCCKKQURMMVV5J47IA2IXRT55YNVYWAFTMCK6DDA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 11:04:49.173129541 (523h35m49.044212576s) [noise=true , address=77.90.31.19:4842 , piece_id=XWXLG22SJWP7SCCKKQURMMVV5J47IA2IXRT55YNVYWAFTMCK6DDA, seg_pos={0 0}, upload=459326]: Long tail cancellation due to optimal threshold Dec 27 11:04:49.173258016 (523h35m49.044341061s) [noise=true , address=45.91.116.6:4973 , piece_id=FJNZX5JAXCYYO6EQXDVI5AN3EMB365FSZGDAQKWOKQMMVBBNQSVA, seg_pos={0 0}, upload=459326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:54886->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:54886->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->45.91.116.6:4973: use of closed network connection Dec 27 11:04:49.173263767 (523h35m49.044346802s) [noise=true , address=45.91.116.6:4973 , piece_id=FJNZX5JAXCYYO6EQXDVI5AN3EMB365FSZGDAQKWOKQMMVBBNQSVA, seg_pos={0 0}, upload=459326]: Long tail cancellation due to optimal threshold Dec 27 11:04:49.173285268 (523h35m49.044368303s) [seg_pos={0 0}, upload=459326]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:04:49.173353248 (523h35m49.044436293s) [upload=459326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:04:49.173357476 (523h35m49.044440511s) [upload=459326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:04:49.468997814 (523h35m49.340080849s) [upload=459326]: Done uploading segments... Dec 27 11:04:49.469017522 (523h35m49.340100557s) [upload=459326]: Upload finished. err: <nil> Duration: 30.002240508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X..] duration: 8.637954754s trace: https://jaeger.dev.storj.io/trace/13dd5a540b9d33f9 FAIL (2025-12-27 11:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A) context deadline exceeded ======LOGS====== 27 11:02:21.590020520 (523h33m21.461103565s) [upload=459292]: Starting upload Dec 27 11:02:21.590098700 (523h33m21.461181735s) [upload=459292]: Done starting upload Dec 27 11:02:21.590281708 (523h33m21.461364753s) [upload=459292]: Uploading segments... Dec 27 11:02:21.590343386 (523h33m21.461426431s) [upload=459292]: Got next segment. Inline: false Dec 27 11:02:21.590377721 (523h33m21.461460756s) [upload=459292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:02:21.590382440 (523h33m21.461465475s) [upload=459292]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:02:21.835621947 (523h33m21.706704983s) [seg_pos={0 0}, upload=459292]: Begin upload segment... Dec 27 11:02:21.835663476 (523h33m21.706746511s) [seg_pos={0 0}, upload=459292]: Done begin upload segment. Dec 27 11:02:21.835699435 (523h33m21.70678247s) [noise=true , address=145.79.227.16:4911 , piece_id=WBGG5VFVWB6VRY7YLN2CD22VMOYD4KC5MO5N3GCJNQG37APSJZRA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.835715866 (523h33m21.706798911s) [noise=true , address=145.79.227.15:4873 , piece_id=PIKN23RU32JX3XCMGJZPYGIINOLLUUIBAGKTK63VAME7KT33UD7A, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836170291 (523h33m21.707253336s) [noise=true , address=45.91.116.10:4083 , piece_id=YUGAZ7KXJOMBSOZ6OBVQER47BOQGWWYZHTQ3PQVYYBTD2AQKGLCA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836462548 (523h33m21.707545593s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836736219 (523h33m21.707819254s) [noise=true , address=77.90.31.14:4663 , piece_id=SJSVSASMBRSGD62YM7UHDRVPLLXTQEC52KY6IB4NTF7UPXCP4HZQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837018827 (523h33m21.708101862s) [noise=true , address=45.91.116.5:4298 , piece_id=EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837298209 (523h33m21.708381244s) [noise=true , address=145.79.227.39:4014 , piece_id=7LWAJJ6HOROGYATKN63XPYHUZ2JI24C3BBVFY5H2TJU6CSR2P45Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837591127 (523h33m21.708674162s) [noise=true , address=145.79.227.42:4122 , piece_id=YBGIK3KKVVYZJ7AVNGLKRE6OICDBOKF7UNZLO4XPJ5VXXPUIDFSQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.838179317 (523h33m21.709262352s) [noise=true , address=45.91.116.7:4011 , piece_id=CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.839765407 (523h33m21.710848452s) [noise=true , address=145.79.227.17:4971 , piece_id=5VSZUA6TZ2HZQEA7SPBAOL4UHPUIOS6FNI754DWUPC4PQXZMVM2Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.841234585 (523h33m21.71231763s) [noise=true , address=145.79.227.38:4979 , piece_id=63XLBNCLV2X4ZMRZMPG6Q3JFWQLGJMLRNP7PHWTGQ6YGTQTVMMYQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.842498942 (523h33m21.713581977s) [noise=true , address=77.90.31.10:4518 , piece_id=JFH735VYMFATJHLWYGEWKRH3YVWQXTSP6BUZEOUAAWKBYNNA2WMQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.843668670 (523h33m21.714751705s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.844938798 (523h33m21.716021833s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.846342992 (523h33m21.717426027s) [noise=true , address=77.90.31.13:4629 , piece_id=JKMKST3XF7XRIJRLJV6354GGU6SYRX5NV6SA4DF44RDJRCJEZOXQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.847877384 (523h33m21.718960419s) [noise=true , address=145.79.227.46:4603 , piece_id=M7ALBOMUL2DNLR63DYGUWHEOSQU2EFHU2EJ3WJAXZKUDPNVIOB7Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:23.565281100 (523h33m23.436364155s) [upload=459292]: Next returned nil segment Dec 27 11:02:23.565318721 (523h33m23.436401766s) [upload=459292]: Waiting for error group managing segments... Dec 27 11:02:23.845237163 (523h33m23.716320208s) [noise=true , address=145.79.227.16:4911 , piece_id=WBGG5VFVWB6VRY7YLN2CD22VMOYD4KC5MO5N3GCJNQG37APSJZRA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.847189240 (523h33m23.718272325s) [noise=true , address=145.79.227.38:4979 , piece_id=63XLBNCLV2X4ZMRZMPG6Q3JFWQLGJMLRNP7PHWTGQ6YGTQTVMMYQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.854410591 (523h33m23.725493636s) [noise=true , address=145.79.227.17:4971 , piece_id=5VSZUA6TZ2HZQEA7SPBAOL4UHPUIOS6FNI754DWUPC4PQXZMVM2Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.865413132 (523h33m23.736496177s) [noise=true , address=145.79.227.39:4014 , piece_id=7LWAJJ6HOROGYATKN63XPYHUZ2JI24C3BBVFY5H2TJU6CSR2P45Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.867133177 (523h33m23.738216212s) [noise=true , address=145.79.227.15:4873 , piece_id=PIKN23RU32JX3XCMGJZPYGIINOLLUUIBAGKTK63VAME7KT33UD7A, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.867462194 (523h33m23.738545239s) [noise=true , address=145.79.227.42:4122 , piece_id=YBGIK3KKVVYZJ7AVNGLKRE6OICDBOKF7UNZLO4XPJ5VXXPUIDFSQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.871714381 (523h33m23.742797476s) [noise=true , address=145.79.227.46:4603 , piece_id=M7ALBOMUL2DNLR63DYGUWHEOSQU2EFHU2EJ3WJAXZKUDPNVIOB7Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.017300251 (523h33m23.888383286s) [noise=true , address=77.90.31.10:4518 , piece_id=JFH735VYMFATJHLWYGEWKRH3YVWQXTSP6BUZEOUAAWKBYNNA2WMQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.086671926 (523h33m23.957754971s) [noise=true , address=45.91.116.7:4011 , piece_id=CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.134788257 (523h33m24.005871303s) [noise=true , address=77.90.31.13:4629 , piece_id=JKMKST3XF7XRIJRLJV6354GGU6SYRX5NV6SA4DF44RDJRCJEZOXQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.155754250 (523h33m24.026837285s) [noise=true , address=45.91.116.5:4298 , piece_id=EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.256389858 (523h33m24.127472893s) [noise=true , address=45.91.116.10:4083 , piece_id=YUGAZ7KXJOMBSOZ6OBVQER47BOQGWWYZHTQ3PQVYYBTD2AQKGLCA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.696486817 (523h33m24.567569852s) [noise=true , address=77.90.31.14:4663 , piece_id=SJSVSASMBRSGD62YM7UHDRVPLLXTQEC52KY6IB4NTF7UPXCP4HZQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.696512716 (523h33m24.567595761s) [seg_pos={0 0}, upload=459292]: Segment reached optimal threshold of 13 pieces. Dec 27 11:02:24.696805193 (523h33m24.567888238s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Dec 27 11:02:24.696814812 (523h33m24.567897857s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.696986509 (523h33m24.568069564s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection Dec 27 11:02:24.697009201 (523h33m24.568092246s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.697129932 (523h33m24.568212977s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 11:02:24.697139830 (523h33m24.568222875s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.697163175 (523h33m24.56824622s) [seg_pos={0 0}, upload=459292]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:02:24.697247656 (523h33m24.568330691s) [upload=459292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:02:24.697254539 (523h33m24.568337574s) [upload=459292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:02:25.000606832 (523h33m24.871689897s) [upload=459292]: Done uploading segments... Dec 27 11:02:25.000635707 (523h33m24.871718763s) [upload=459292]: Upload finished. err: <nil> Duration: 8.637974382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X.] duration: 5.938587103s trace: https://jaeger.dev.storj.io/trace/13dd5a540b9d33f9 FAIL (2025-12-27 11:03:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A) context deadline exceeded ======LOGS====== 27 11:02:21.590020520 (523h33m21.461103565s) [upload=459292]: Starting upload Dec 27 11:02:21.590098700 (523h33m21.461181735s) [upload=459292]: Done starting upload Dec 27 11:02:21.590281708 (523h33m21.461364753s) [upload=459292]: Uploading segments... Dec 27 11:02:21.590343386 (523h33m21.461426431s) [upload=459292]: Got next segment. Inline: false Dec 27 11:02:21.590377721 (523h33m21.461460756s) [upload=459292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:02:21.590382440 (523h33m21.461465475s) [upload=459292]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:02:21.835621947 (523h33m21.706704983s) [seg_pos={0 0}, upload=459292]: Begin upload segment... Dec 27 11:02:21.835663476 (523h33m21.706746511s) [seg_pos={0 0}, upload=459292]: Done begin upload segment. Dec 27 11:02:21.835699435 (523h33m21.70678247s) [noise=true , address=145.79.227.16:4911 , piece_id=WBGG5VFVWB6VRY7YLN2CD22VMOYD4KC5MO5N3GCJNQG37APSJZRA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.835715866 (523h33m21.706798911s) [noise=true , address=145.79.227.15:4873 , piece_id=PIKN23RU32JX3XCMGJZPYGIINOLLUUIBAGKTK63VAME7KT33UD7A, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836170291 (523h33m21.707253336s) [noise=true , address=45.91.116.10:4083 , piece_id=YUGAZ7KXJOMBSOZ6OBVQER47BOQGWWYZHTQ3PQVYYBTD2AQKGLCA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836462548 (523h33m21.707545593s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836736219 (523h33m21.707819254s) [noise=true , address=77.90.31.14:4663 , piece_id=SJSVSASMBRSGD62YM7UHDRVPLLXTQEC52KY6IB4NTF7UPXCP4HZQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837018827 (523h33m21.708101862s) [noise=true , address=45.91.116.5:4298 , piece_id=EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837298209 (523h33m21.708381244s) [noise=true , address=145.79.227.39:4014 , piece_id=7LWAJJ6HOROGYATKN63XPYHUZ2JI24C3BBVFY5H2TJU6CSR2P45Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837591127 (523h33m21.708674162s) [noise=true , address=145.79.227.42:4122 , piece_id=YBGIK3KKVVYZJ7AVNGLKRE6OICDBOKF7UNZLO4XPJ5VXXPUIDFSQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.838179317 (523h33m21.709262352s) [noise=true , address=45.91.116.7:4011 , piece_id=CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.839765407 (523h33m21.710848452s) [noise=true , address=145.79.227.17:4971 , piece_id=5VSZUA6TZ2HZQEA7SPBAOL4UHPUIOS6FNI754DWUPC4PQXZMVM2Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.841234585 (523h33m21.71231763s) [noise=true , address=145.79.227.38:4979 , piece_id=63XLBNCLV2X4ZMRZMPG6Q3JFWQLGJMLRNP7PHWTGQ6YGTQTVMMYQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.842498942 (523h33m21.713581977s) [noise=true , address=77.90.31.10:4518 , piece_id=JFH735VYMFATJHLWYGEWKRH3YVWQXTSP6BUZEOUAAWKBYNNA2WMQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.843668670 (523h33m21.714751705s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.844938798 (523h33m21.716021833s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.846342992 (523h33m21.717426027s) [noise=true , address=77.90.31.13:4629 , piece_id=JKMKST3XF7XRIJRLJV6354GGU6SYRX5NV6SA4DF44RDJRCJEZOXQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.847877384 (523h33m21.718960419s) [noise=true , address=145.79.227.46:4603 , piece_id=M7ALBOMUL2DNLR63DYGUWHEOSQU2EFHU2EJ3WJAXZKUDPNVIOB7Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:23.565281100 (523h33m23.436364155s) [upload=459292]: Next returned nil segment Dec 27 11:02:23.565318721 (523h33m23.436401766s) [upload=459292]: Waiting for error group managing segments... Dec 27 11:02:23.845237163 (523h33m23.716320208s) [noise=true , address=145.79.227.16:4911 , piece_id=WBGG5VFVWB6VRY7YLN2CD22VMOYD4KC5MO5N3GCJNQG37APSJZRA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.847189240 (523h33m23.718272325s) [noise=true , address=145.79.227.38:4979 , piece_id=63XLBNCLV2X4ZMRZMPG6Q3JFWQLGJMLRNP7PHWTGQ6YGTQTVMMYQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.854410591 (523h33m23.725493636s) [noise=true , address=145.79.227.17:4971 , piece_id=5VSZUA6TZ2HZQEA7SPBAOL4UHPUIOS6FNI754DWUPC4PQXZMVM2Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.865413132 (523h33m23.736496177s) [noise=true , address=145.79.227.39:4014 , piece_id=7LWAJJ6HOROGYATKN63XPYHUZ2JI24C3BBVFY5H2TJU6CSR2P45Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.867133177 (523h33m23.738216212s) [noise=true , address=145.79.227.15:4873 , piece_id=PIKN23RU32JX3XCMGJZPYGIINOLLUUIBAGKTK63VAME7KT33UD7A, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.867462194 (523h33m23.738545239s) [noise=true , address=145.79.227.42:4122 , piece_id=YBGIK3KKVVYZJ7AVNGLKRE6OICDBOKF7UNZLO4XPJ5VXXPUIDFSQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.871714381 (523h33m23.742797476s) [noise=true , address=145.79.227.46:4603 , piece_id=M7ALBOMUL2DNLR63DYGUWHEOSQU2EFHU2EJ3WJAXZKUDPNVIOB7Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.017300251 (523h33m23.888383286s) [noise=true , address=77.90.31.10:4518 , piece_id=JFH735VYMFATJHLWYGEWKRH3YVWQXTSP6BUZEOUAAWKBYNNA2WMQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.086671926 (523h33m23.957754971s) [noise=true , address=45.91.116.7:4011 , piece_id=CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.134788257 (523h33m24.005871303s) [noise=true , address=77.90.31.13:4629 , piece_id=JKMKST3XF7XRIJRLJV6354GGU6SYRX5NV6SA4DF44RDJRCJEZOXQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.155754250 (523h33m24.026837285s) [noise=true , address=45.91.116.5:4298 , piece_id=EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.256389858 (523h33m24.127472893s) [noise=true , address=45.91.116.10:4083 , piece_id=YUGAZ7KXJOMBSOZ6OBVQER47BOQGWWYZHTQ3PQVYYBTD2AQKGLCA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.696486817 (523h33m24.567569852s) [noise=true , address=77.90.31.14:4663 , piece_id=SJSVSASMBRSGD62YM7UHDRVPLLXTQEC52KY6IB4NTF7UPXCP4HZQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.696512716 (523h33m24.567595761s) [seg_pos={0 0}, upload=459292]: Segment reached optimal threshold of 13 pieces. Dec 27 11:02:24.696805193 (523h33m24.567888238s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Dec 27 11:02:24.696814812 (523h33m24.567897857s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.696986509 (523h33m24.568069564s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection Dec 27 11:02:24.697009201 (523h33m24.568092246s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.697129932 (523h33m24.568212977s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 11:02:24.697139830 (523h33m24.568222875s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.697163175 (523h33m24.56824622s) [seg_pos={0 0}, upload=459292]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:02:24.697247656 (523h33m24.568330691s) [upload=459292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:02:24.697254539 (523h33m24.568337574s) [upload=459292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:02:25.000606832 (523h33m24.871689897s) [upload=459292]: Done uploading segments... Dec 27 11:02:25.000635707 (523h33m24.871718763s) [upload=459292]: Upload finished. err: <nil> Duration: 5.938619335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.X] duration: 30.003011061s trace: https://jaeger.dev.storj.io/trace/13dd5a540b9d33f9 FAIL (2025-12-27 11:02:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A) context deadline exceeded ======LOGS====== 27 11:02:21.590020520 (523h33m21.461103565s) [upload=459292]: Starting upload Dec 27 11:02:21.590098700 (523h33m21.461181735s) [upload=459292]: Done starting upload Dec 27 11:02:21.590281708 (523h33m21.461364753s) [upload=459292]: Uploading segments... Dec 27 11:02:21.590343386 (523h33m21.461426431s) [upload=459292]: Got next segment. Inline: false Dec 27 11:02:21.590377721 (523h33m21.461460756s) [upload=459292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:02:21.590382440 (523h33m21.461465475s) [upload=459292]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:02:21.835621947 (523h33m21.706704983s) [seg_pos={0 0}, upload=459292]: Begin upload segment... Dec 27 11:02:21.835663476 (523h33m21.706746511s) [seg_pos={0 0}, upload=459292]: Done begin upload segment. Dec 27 11:02:21.835699435 (523h33m21.70678247s) [noise=true , address=145.79.227.16:4911 , piece_id=WBGG5VFVWB6VRY7YLN2CD22VMOYD4KC5MO5N3GCJNQG37APSJZRA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.835715866 (523h33m21.706798911s) [noise=true , address=145.79.227.15:4873 , piece_id=PIKN23RU32JX3XCMGJZPYGIINOLLUUIBAGKTK63VAME7KT33UD7A, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836170291 (523h33m21.707253336s) [noise=true , address=45.91.116.10:4083 , piece_id=YUGAZ7KXJOMBSOZ6OBVQER47BOQGWWYZHTQ3PQVYYBTD2AQKGLCA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836462548 (523h33m21.707545593s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.836736219 (523h33m21.707819254s) [noise=true , address=77.90.31.14:4663 , piece_id=SJSVSASMBRSGD62YM7UHDRVPLLXTQEC52KY6IB4NTF7UPXCP4HZQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837018827 (523h33m21.708101862s) [noise=true , address=45.91.116.5:4298 , piece_id=EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837298209 (523h33m21.708381244s) [noise=true , address=145.79.227.39:4014 , piece_id=7LWAJJ6HOROGYATKN63XPYHUZ2JI24C3BBVFY5H2TJU6CSR2P45Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.837591127 (523h33m21.708674162s) [noise=true , address=145.79.227.42:4122 , piece_id=YBGIK3KKVVYZJ7AVNGLKRE6OICDBOKF7UNZLO4XPJ5VXXPUIDFSQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.838179317 (523h33m21.709262352s) [noise=true , address=45.91.116.7:4011 , piece_id=CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.839765407 (523h33m21.710848452s) [noise=true , address=145.79.227.17:4971 , piece_id=5VSZUA6TZ2HZQEA7SPBAOL4UHPUIOS6FNI754DWUPC4PQXZMVM2Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.841234585 (523h33m21.71231763s) [noise=true , address=145.79.227.38:4979 , piece_id=63XLBNCLV2X4ZMRZMPG6Q3JFWQLGJMLRNP7PHWTGQ6YGTQTVMMYQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.842498942 (523h33m21.713581977s) [noise=true , address=77.90.31.10:4518 , piece_id=JFH735VYMFATJHLWYGEWKRH3YVWQXTSP6BUZEOUAAWKBYNNA2WMQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.843668670 (523h33m21.714751705s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.844938798 (523h33m21.716021833s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.846342992 (523h33m21.717426027s) [noise=true , address=77.90.31.13:4629 , piece_id=JKMKST3XF7XRIJRLJV6354GGU6SYRX5NV6SA4DF44RDJRCJEZOXQ, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:21.847877384 (523h33m21.718960419s) [noise=true , address=145.79.227.46:4603 , piece_id=M7ALBOMUL2DNLR63DYGUWHEOSQU2EFHU2EJ3WJAXZKUDPNVIOB7Q, seg_pos={0 0}, upload=459292]: Uploading piece... Dec 27 11:02:23.565281100 (523h33m23.436364155s) [upload=459292]: Next returned nil segment Dec 27 11:02:23.565318721 (523h33m23.436401766s) [upload=459292]: Waiting for error group managing segments... Dec 27 11:02:23.845237163 (523h33m23.716320208s) [noise=true , address=145.79.227.16:4911 , piece_id=WBGG5VFVWB6VRY7YLN2CD22VMOYD4KC5MO5N3GCJNQG37APSJZRA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.847189240 (523h33m23.718272325s) [noise=true , address=145.79.227.38:4979 , piece_id=63XLBNCLV2X4ZMRZMPG6Q3JFWQLGJMLRNP7PHWTGQ6YGTQTVMMYQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.854410591 (523h33m23.725493636s) [noise=true , address=145.79.227.17:4971 , piece_id=5VSZUA6TZ2HZQEA7SPBAOL4UHPUIOS6FNI754DWUPC4PQXZMVM2Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.865413132 (523h33m23.736496177s) [noise=true , address=145.79.227.39:4014 , piece_id=7LWAJJ6HOROGYATKN63XPYHUZ2JI24C3BBVFY5H2TJU6CSR2P45Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.867133177 (523h33m23.738216212s) [noise=true , address=145.79.227.15:4873 , piece_id=PIKN23RU32JX3XCMGJZPYGIINOLLUUIBAGKTK63VAME7KT33UD7A, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.867462194 (523h33m23.738545239s) [noise=true , address=145.79.227.42:4122 , piece_id=YBGIK3KKVVYZJ7AVNGLKRE6OICDBOKF7UNZLO4XPJ5VXXPUIDFSQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:23.871714381 (523h33m23.742797476s) [noise=true , address=145.79.227.46:4603 , piece_id=M7ALBOMUL2DNLR63DYGUWHEOSQU2EFHU2EJ3WJAXZKUDPNVIOB7Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.017300251 (523h33m23.888383286s) [noise=true , address=77.90.31.10:4518 , piece_id=JFH735VYMFATJHLWYGEWKRH3YVWQXTSP6BUZEOUAAWKBYNNA2WMQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.086671926 (523h33m23.957754971s) [noise=true , address=45.91.116.7:4011 , piece_id=CGLDZ64QR2ITIE75HVRPRQ2EHF4DMLLRLN674ID7RFIPOE6XZF5Q, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.134788257 (523h33m24.005871303s) [noise=true , address=77.90.31.13:4629 , piece_id=JKMKST3XF7XRIJRLJV6354GGU6SYRX5NV6SA4DF44RDJRCJEZOXQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.155754250 (523h33m24.026837285s) [noise=true , address=45.91.116.5:4298 , piece_id=EUEU3UPF24JAT3SZOGGCOCPNS3TCXK2KLXOD3HBFTFBTK5CZQQ7A, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.256389858 (523h33m24.127472893s) [noise=true , address=45.91.116.10:4083 , piece_id=YUGAZ7KXJOMBSOZ6OBVQER47BOQGWWYZHTQ3PQVYYBTD2AQKGLCA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.696486817 (523h33m24.567569852s) [noise=true , address=77.90.31.14:4663 , piece_id=SJSVSASMBRSGD62YM7UHDRVPLLXTQEC52KY6IB4NTF7UPXCP4HZQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: <nil> Dec 27 11:02:24.696512716 (523h33m24.567595761s) [seg_pos={0 0}, upload=459292]: Segment reached optimal threshold of 13 pieces. Dec 27 11:02:24.696805193 (523h33m24.567888238s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Dec 27 11:02:24.696814812 (523h33m24.567897857s) [noise=true , address=145.79.227.8:4331 , piece_id=76XAL4Q4D4NVKD3S2BTRHZJRZCHS7FYOWZRWIQHOSUMVQDP3YLTA, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.696986509 (523h33m24.568069564s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45246->77.90.31.12:4589: use of closed network connection Dec 27 11:02:24.697009201 (523h33m24.568092246s) [noise=true , address=77.90.31.12:4589 , piece_id=HD25X4TQTFCOX2W7FYIJJPDBHOEZRYKOEMLYYGSDJQCPNXNCJVUQ, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.697129932 (523h33m24.568212977s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 27 11:02:24.697139830 (523h33m24.568222875s) [noise=true , address=145.79.227.33:4870 , piece_id=6LHUZQDKYU6QYAIIOF7Z3TFPJGCHTYXDS7ESJ7SCCQXWNIMXTGPQ, seg_pos={0 0}, upload=459292]: Long tail cancellation due to optimal threshold Dec 27 11:02:24.697163175 (523h33m24.56824622s) [seg_pos={0 0}, upload=459292]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:02:24.697247656 (523h33m24.568330691s) [upload=459292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:02:24.697254539 (523h33m24.568337574s) [upload=459292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:02:25.000606832 (523h33m24.871689897s) [upload=459292]: Done uploading segments... Dec 27 11:02:25.000635707 (523h33m24.871718763s) [upload=459292]: Upload finished. err: <nil> Duration: 30.00303757s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX.] duration: 5.892026866s trace: https://jaeger.dev.storj.io/trace/5b619dcd21d41875 FAIL (2025-12-27 11:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 11:01:03.677024956 (523h32m3.548108001s) [upload=459274]: Starting upload Dec 27 11:01:03.677121210 (523h32m3.548204265s) [upload=459274]: Done starting upload Dec 27 11:01:03.677198386 (523h32m3.548281432s) [upload=459274]: Uploading segments... Dec 27 11:01:03.677404108 (523h32m3.548487153s) [upload=459274]: Got next segment. Inline: false Dec 27 11:01:03.677437582 (523h32m3.548520627s) [upload=459274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:01:03.677446268 (523h32m3.548529313s) [upload=459274]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:01:03.923621688 (523h32m3.794704733s) [seg_pos={0 0}, upload=459274]: Begin upload segment... Dec 27 11:01:03.923686572 (523h32m3.794769607s) [seg_pos={0 0}, upload=459274]: Done begin upload segment. Dec 27 11:01:03.923751806 (523h32m3.794834841s) [noise=true , address=145.79.227.38:4977 , piece_id=MZ6PZAIKHYAOC2Z3FXB2LYNUS5VRNVHZZCCZZV3VYEC7UTWWS5JQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.923774068 (523h32m3.794857123s) [noise=true , address=145.79.227.4:4186 , piece_id=MJI2VJFWSJMDJSJ7EO5P4T5IEYNC5CGDHF63P4B5BIIMFECONWFQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.924597346 (523h32m3.795680381s) [noise=true , address=145.79.227.16:4910 , piece_id=2H3QUDK3D6ZOSMNR2UVMOU5X2QGBPKFWDJZD5QUJQF5OYDY6Z7SA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.925199993 (523h32m3.796283068s) [noise=true , address=145.79.227.46:4603 , piece_id=RW5FGOHCYDHYDFMRMKJLXZRQOWX4E36JMLCNVVVZUNT7WUTRQMPA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.925755872 (523h32m3.796838917s) [noise=true , address=145.79.227.17:4972 , piece_id=Y6X3KQIM5SYSEFEFDCASNSNJQ7OHDELQMWQ25TSTPON5RAVTP4MA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.926486573 (523h32m3.797569608s) [noise=true , address=145.79.227.52:4445 , piece_id=S7FA7HYDWSUID4T4IWP2WVNC7RONFG6WIT5DRETXWUN5PQMMIM5A, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.927074402 (523h32m3.798157447s) [noise=true , address=145.79.227.11:4730 , piece_id=GXZ673RM6IJNBELNLJ6XFQKSQ76WQ5S3IP73WZN5AERYWFZRRMRQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.928727841 (523h32m3.799810886s) [noise=true , address=77.90.31.11:4556 , piece_id=PLR2IERVINBSDX3VPH2OEY2WZAZDX3RUIN5Y7TTSIMJRBV7UUKGA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.930140781 (523h32m3.801223826s) [noise=true , address=145.79.227.15:4876 , piece_id=2PFZ4V6QLKPIOSCLXIFNC7OEBUPAVWHSEXFMOZ5UTCKDPITV4G6A, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.930346042 (523h32m3.801429097s) [noise=true , address=45.91.116.7:4012 , piece_id=SURRTABI63EDGMX55ZCKKSK5HEHT57ZYVYUN6S7HEGOUO5FC6F4A, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.933877577 (523h32m3.804960622s) [noise=true , address=77.90.31.12:4590 , piece_id=VGDCYIPEB74BXZVR2QZPXKM4XOWXYWLNLJYSMPEFRGRR74CB53WA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.936262709 (523h32m3.807345754s) [noise=true , address=145.79.227.2:4113 , piece_id=GFUF3UCP6DUDLM37KBG4I45GSHHVYT5QGAGQFTG23IJPEUB7A7AQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.937605796 (523h32m3.808688832s) [noise=true , address=145.79.227.3:4149 , piece_id=7POTVECORM7G2XM5CDDXNKZMOMOJPHXFKTEXCXJ6LPSMIMIRDVUA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.938986857 (523h32m3.810069902s) [noise=true , address=77.90.31.14:4664 , piece_id=TMFTWGKL53JGE5R6AUHN3GYVJPQTYUA6ET7W2GNJIIDPW2D5V6DQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.940389076 (523h32m3.811472121s) [noise=true , address=45.91.116.8:4046 , piece_id=RYWMGPHUWV44BNANT3HIAIHJ62XC6YJMB4F6IDQQDJAXH3HEYL5Q, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.942274667 (523h32m3.813357712s) [noise=true , address=77.90.31.13:4629 , piece_id=CW6TV3MAPC2WFKYOG6CMHCKU3JAKHJY22AJLJ6MD7CGHYNS3WLMQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:05.664923151 (523h32m5.536006196s) [upload=459274]: Next returned nil segment Dec 27 11:01:05.664938740 (523h32m5.536021775s) [upload=459274]: Waiting for error group managing segments... Dec 27 11:01:05.951143623 (523h32m5.822226668s) [noise=true , address=145.79.227.4:4186 , piece_id=MJI2VJFWSJMDJSJ7EO5P4T5IEYNC5CGDHF63P4B5BIIMFECONWFQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.954928430 (523h32m5.826011475s) [noise=true , address=145.79.227.16:4910 , piece_id=2H3QUDK3D6ZOSMNR2UVMOU5X2QGBPKFWDJZD5QUJQF5OYDY6Z7SA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.955835758 (523h32m5.826918803s) [noise=true , address=145.79.227.38:4977 , piece_id=MZ6PZAIKHYAOC2Z3FXB2LYNUS5VRNVHZZCCZZV3VYEC7UTWWS5JQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.958778552 (523h32m5.829861587s) [noise=true , address=145.79.227.3:4149 , piece_id=7POTVECORM7G2XM5CDDXNKZMOMOJPHXFKTEXCXJ6LPSMIMIRDVUA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.959953668 (523h32m5.831036703s) [noise=true , address=145.79.227.2:4113 , piece_id=GFUF3UCP6DUDLM37KBG4I45GSHHVYT5QGAGQFTG23IJPEUB7A7AQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.963089940 (523h32m5.834172985s) [noise=true , address=145.79.227.15:4876 , piece_id=2PFZ4V6QLKPIOSCLXIFNC7OEBUPAVWHSEXFMOZ5UTCKDPITV4G6A, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.970703067 (523h32m5.841786112s) [noise=true , address=145.79.227.46:4603 , piece_id=RW5FGOHCYDHYDFMRMKJLXZRQOWX4E36JMLCNVVVZUNT7WUTRQMPA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.971684837 (523h32m5.842767882s) [noise=true , address=145.79.227.17:4972 , piece_id=Y6X3KQIM5SYSEFEFDCASNSNJQ7OHDELQMWQ25TSTPON5RAVTP4MA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.987045716 (523h32m5.858128761s) [noise=true , address=145.79.227.52:4445 , piece_id=S7FA7HYDWSUID4T4IWP2WVNC7RONFG6WIT5DRETXWUN5PQMMIM5A, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:06.160182910 (523h32m6.031265955s) [noise=true , address=45.91.116.7:4012 , piece_id=SURRTABI63EDGMX55ZCKKSK5HEHT57ZYVYUN6S7HEGOUO5FC6F4A, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:06.315235265 (523h32m6.1863183s) [noise=true , address=145.79.227.11:4730 , piece_id=GXZ673RM6IJNBELNLJ6XFQKSQ76WQ5S3IP73WZN5AERYWFZRRMRQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:06.887610003 (523h32m6.758693048s) [noise=true , address=77.90.31.12:4590 , piece_id=VGDCYIPEB74BXZVR2QZPXKM4XOWXYWLNLJYSMPEFRGRR74CB53WA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:07.035430809 (523h32m6.906513864s) [noise=true , address=77.90.31.13:4629 , piece_id=CW6TV3MAPC2WFKYOG6CMHCKU3JAKHJY22AJLJ6MD7CGHYNS3WLMQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:07.035459834 (523h32m6.906542869s) [seg_pos={0 0}, upload=459274]: Segment reached optimal threshold of 13 pieces. Dec 27 11:01:07.035880755 (523h32m6.906963831s) [noise=true , address=77.90.31.14:4664 , piece_id=TMFTWGKL53JGE5R6AUHN3GYVJPQTYUA6ET7W2GNJIIDPW2D5V6DQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 11:01:07.035896516 (523h32m6.906979551s) [noise=true , address=77.90.31.14:4664 , piece_id=TMFTWGKL53JGE5R6AUHN3GYVJPQTYUA6ET7W2GNJIIDPW2D5V6DQ, seg_pos={0 0}, upload=459274]: Long tail cancellation due to optimal threshold Dec 27 11:01:07.036160769 (523h32m6.907243814s) [noise=true , address=45.91.116.8:4046 , piece_id=RYWMGPHUWV44BNANT3HIAIHJ62XC6YJMB4F6IDQQDJAXH3HEYL5Q, seg_pos={0 0}, upload=459274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 11:01:07.036199113 (523h32m6.907282158s) [noise=true , address=45.91.116.8:4046 , piece_id=RYWMGPHUWV44BNANT3HIAIHJ62XC6YJMB4F6IDQQDJAXH3HEYL5Q, seg_pos={0 0}, upload=459274]: Long tail cancellation due to optimal threshold Dec 27 11:01:07.036249618 (523h32m6.907332653s) [noise=true , address=77.90.31.11:4556 , piece_id=PLR2IERVINBSDX3VPH2OEY2WZAZDX3RUIN5Y7TTSIMJRBV7UUKGA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 11:01:07.036262422 (523h32m6.907345467s) [noise=true , address=77.90.31.11:4556 , piece_id=PLR2IERVINBSDX3VPH2OEY2WZAZDX3RUIN5Y7TTSIMJRBV7UUKGA, seg_pos={0 0}, upload=459274]: Long tail cancellation due to optimal threshold Dec 27 11:01:07.036288993 (523h32m6.907372038s) [seg_pos={0 0}, upload=459274]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:01:07.036384285 (523h32m6.90746733s) [upload=459274]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:01:07.036392039 (523h32m6.907475084s) [upload=459274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:01:07.323944227 (523h32m7.195027272s) [upload=459274]: Done uploading segments... Dec 27 11:01:07.323964937 (523h32m7.195047982s) [upload=459274]: Upload finished. err: <nil> Duration: 5.892042686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXXX] duration: 30.003278894s trace: https://jaeger.dev.storj.io/trace/5b619dcd21d41875 FAIL (2025-12-27 11:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 11:01:03.677024956 (523h32m3.548108001s) [upload=459274]: Starting upload Dec 27 11:01:03.677121210 (523h32m3.548204265s) [upload=459274]: Done starting upload Dec 27 11:01:03.677198386 (523h32m3.548281432s) [upload=459274]: Uploading segments... Dec 27 11:01:03.677404108 (523h32m3.548487153s) [upload=459274]: Got next segment. Inline: false Dec 27 11:01:03.677437582 (523h32m3.548520627s) [upload=459274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:01:03.677446268 (523h32m3.548529313s) [upload=459274]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:01:03.923621688 (523h32m3.794704733s) [seg_pos={0 0}, upload=459274]: Begin upload segment... Dec 27 11:01:03.923686572 (523h32m3.794769607s) [seg_pos={0 0}, upload=459274]: Done begin upload segment. Dec 27 11:01:03.923751806 (523h32m3.794834841s) [noise=true , address=145.79.227.38:4977 , piece_id=MZ6PZAIKHYAOC2Z3FXB2LYNUS5VRNVHZZCCZZV3VYEC7UTWWS5JQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.923774068 (523h32m3.794857123s) [noise=true , address=145.79.227.4:4186 , piece_id=MJI2VJFWSJMDJSJ7EO5P4T5IEYNC5CGDHF63P4B5BIIMFECONWFQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.924597346 (523h32m3.795680381s) [noise=true , address=145.79.227.16:4910 , piece_id=2H3QUDK3D6ZOSMNR2UVMOU5X2QGBPKFWDJZD5QUJQF5OYDY6Z7SA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.925199993 (523h32m3.796283068s) [noise=true , address=145.79.227.46:4603 , piece_id=RW5FGOHCYDHYDFMRMKJLXZRQOWX4E36JMLCNVVVZUNT7WUTRQMPA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.925755872 (523h32m3.796838917s) [noise=true , address=145.79.227.17:4972 , piece_id=Y6X3KQIM5SYSEFEFDCASNSNJQ7OHDELQMWQ25TSTPON5RAVTP4MA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.926486573 (523h32m3.797569608s) [noise=true , address=145.79.227.52:4445 , piece_id=S7FA7HYDWSUID4T4IWP2WVNC7RONFG6WIT5DRETXWUN5PQMMIM5A, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.927074402 (523h32m3.798157447s) [noise=true , address=145.79.227.11:4730 , piece_id=GXZ673RM6IJNBELNLJ6XFQKSQ76WQ5S3IP73WZN5AERYWFZRRMRQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.928727841 (523h32m3.799810886s) [noise=true , address=77.90.31.11:4556 , piece_id=PLR2IERVINBSDX3VPH2OEY2WZAZDX3RUIN5Y7TTSIMJRBV7UUKGA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.930140781 (523h32m3.801223826s) [noise=true , address=145.79.227.15:4876 , piece_id=2PFZ4V6QLKPIOSCLXIFNC7OEBUPAVWHSEXFMOZ5UTCKDPITV4G6A, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.930346042 (523h32m3.801429097s) [noise=true , address=45.91.116.7:4012 , piece_id=SURRTABI63EDGMX55ZCKKSK5HEHT57ZYVYUN6S7HEGOUO5FC6F4A, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.933877577 (523h32m3.804960622s) [noise=true , address=77.90.31.12:4590 , piece_id=VGDCYIPEB74BXZVR2QZPXKM4XOWXYWLNLJYSMPEFRGRR74CB53WA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.936262709 (523h32m3.807345754s) [noise=true , address=145.79.227.2:4113 , piece_id=GFUF3UCP6DUDLM37KBG4I45GSHHVYT5QGAGQFTG23IJPEUB7A7AQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.937605796 (523h32m3.808688832s) [noise=true , address=145.79.227.3:4149 , piece_id=7POTVECORM7G2XM5CDDXNKZMOMOJPHXFKTEXCXJ6LPSMIMIRDVUA, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.938986857 (523h32m3.810069902s) [noise=true , address=77.90.31.14:4664 , piece_id=TMFTWGKL53JGE5R6AUHN3GYVJPQTYUA6ET7W2GNJIIDPW2D5V6DQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.940389076 (523h32m3.811472121s) [noise=true , address=45.91.116.8:4046 , piece_id=RYWMGPHUWV44BNANT3HIAIHJ62XC6YJMB4F6IDQQDJAXH3HEYL5Q, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:03.942274667 (523h32m3.813357712s) [noise=true , address=77.90.31.13:4629 , piece_id=CW6TV3MAPC2WFKYOG6CMHCKU3JAKHJY22AJLJ6MD7CGHYNS3WLMQ, seg_pos={0 0}, upload=459274]: Uploading piece... Dec 27 11:01:05.664923151 (523h32m5.536006196s) [upload=459274]: Next returned nil segment Dec 27 11:01:05.664938740 (523h32m5.536021775s) [upload=459274]: Waiting for error group managing segments... Dec 27 11:01:05.951143623 (523h32m5.822226668s) [noise=true , address=145.79.227.4:4186 , piece_id=MJI2VJFWSJMDJSJ7EO5P4T5IEYNC5CGDHF63P4B5BIIMFECONWFQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.954928430 (523h32m5.826011475s) [noise=true , address=145.79.227.16:4910 , piece_id=2H3QUDK3D6ZOSMNR2UVMOU5X2QGBPKFWDJZD5QUJQF5OYDY6Z7SA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.955835758 (523h32m5.826918803s) [noise=true , address=145.79.227.38:4977 , piece_id=MZ6PZAIKHYAOC2Z3FXB2LYNUS5VRNVHZZCCZZV3VYEC7UTWWS5JQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.958778552 (523h32m5.829861587s) [noise=true , address=145.79.227.3:4149 , piece_id=7POTVECORM7G2XM5CDDXNKZMOMOJPHXFKTEXCXJ6LPSMIMIRDVUA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.959953668 (523h32m5.831036703s) [noise=true , address=145.79.227.2:4113 , piece_id=GFUF3UCP6DUDLM37KBG4I45GSHHVYT5QGAGQFTG23IJPEUB7A7AQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.963089940 (523h32m5.834172985s) [noise=true , address=145.79.227.15:4876 , piece_id=2PFZ4V6QLKPIOSCLXIFNC7OEBUPAVWHSEXFMOZ5UTCKDPITV4G6A, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.970703067 (523h32m5.841786112s) [noise=true , address=145.79.227.46:4603 , piece_id=RW5FGOHCYDHYDFMRMKJLXZRQOWX4E36JMLCNVVVZUNT7WUTRQMPA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.971684837 (523h32m5.842767882s) [noise=true , address=145.79.227.17:4972 , piece_id=Y6X3KQIM5SYSEFEFDCASNSNJQ7OHDELQMWQ25TSTPON5RAVTP4MA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:05.987045716 (523h32m5.858128761s) [noise=true , address=145.79.227.52:4445 , piece_id=S7FA7HYDWSUID4T4IWP2WVNC7RONFG6WIT5DRETXWUN5PQMMIM5A, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:06.160182910 (523h32m6.031265955s) [noise=true , address=45.91.116.7:4012 , piece_id=SURRTABI63EDGMX55ZCKKSK5HEHT57ZYVYUN6S7HEGOUO5FC6F4A, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:06.315235265 (523h32m6.1863183s) [noise=true , address=145.79.227.11:4730 , piece_id=GXZ673RM6IJNBELNLJ6XFQKSQ76WQ5S3IP73WZN5AERYWFZRRMRQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:06.887610003 (523h32m6.758693048s) [noise=true , address=77.90.31.12:4590 , piece_id=VGDCYIPEB74BXZVR2QZPXKM4XOWXYWLNLJYSMPEFRGRR74CB53WA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:07.035430809 (523h32m6.906513864s) [noise=true , address=77.90.31.13:4629 , piece_id=CW6TV3MAPC2WFKYOG6CMHCKU3JAKHJY22AJLJ6MD7CGHYNS3WLMQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: <nil> Dec 27 11:01:07.035459834 (523h32m6.906542869s) [seg_pos={0 0}, upload=459274]: Segment reached optimal threshold of 13 pieces. Dec 27 11:01:07.035880755 (523h32m6.906963831s) [noise=true , address=77.90.31.14:4664 , piece_id=TMFTWGKL53JGE5R6AUHN3GYVJPQTYUA6ET7W2GNJIIDPW2D5V6DQ, seg_pos={0 0}, upload=459274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 11:01:07.035896516 (523h32m6.906979551s) [noise=true , address=77.90.31.14:4664 , piece_id=TMFTWGKL53JGE5R6AUHN3GYVJPQTYUA6ET7W2GNJIIDPW2D5V6DQ, seg_pos={0 0}, upload=459274]: Long tail cancellation due to optimal threshold Dec 27 11:01:07.036160769 (523h32m6.907243814s) [noise=true , address=45.91.116.8:4046 , piece_id=RYWMGPHUWV44BNANT3HIAIHJ62XC6YJMB4F6IDQQDJAXH3HEYL5Q, seg_pos={0 0}, upload=459274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 11:01:07.036199113 (523h32m6.907282158s) [noise=true , address=45.91.116.8:4046 , piece_id=RYWMGPHUWV44BNANT3HIAIHJ62XC6YJMB4F6IDQQDJAXH3HEYL5Q, seg_pos={0 0}, upload=459274]: Long tail cancellation due to optimal threshold Dec 27 11:01:07.036249618 (523h32m6.907332653s) [noise=true , address=77.90.31.11:4556 , piece_id=PLR2IERVINBSDX3VPH2OEY2WZAZDX3RUIN5Y7TTSIMJRBV7UUKGA, seg_pos={0 0}, upload=459274]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 11:01:07.036262422 (523h32m6.907345467s) [noise=true , address=77.90.31.11:4556 , piece_id=PLR2IERVINBSDX3VPH2OEY2WZAZDX3RUIN5Y7TTSIMJRBV7UUKGA, seg_pos={0 0}, upload=459274]: Long tail cancellation due to optimal threshold Dec 27 11:01:07.036288993 (523h32m6.907372038s) [seg_pos={0 0}, upload=459274]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:01:07.036384285 (523h32m6.90746733s) [upload=459274]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:01:07.036392039 (523h32m6.907475084s) [upload=459274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:01:07.323944227 (523h32m7.195027272s) [upload=459274]: Done uploading segments... Dec 27 11:01:07.323964937 (523h32m7.195047982s) [upload=459274]: Upload finished. err: <nil> Duration: 30.003305204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XXX] duration: 30.002619713s trace: https://jaeger.dev.storj.io/trace/7b9e2f732cc4d201 FAIL (2025-12-27 11:00:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: IWVAB3U63NFNI3OCSZZ2SQAUFWJYB7QQRRR4BIIQJTX3ZUT3GU5Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: RQ7NU7SBPN3VBQKP232D2VWUTQDAGBHC52MQAQEOLF3734H5GMXQ) context deadline exceeded ======LOGS====== Dec 27 11:00:00.312468403 (523h31m0.183551448s) [upload=459258]: Starting upload Dec 27 11:00:00.312523567 (523h31m0.183606612s) [upload=459258]: Done starting upload Dec 27 11:00:00.312731433 (523h31m0.183814498s) [upload=459258]: Uploading segments... Dec 27 11:00:00.312904823 (523h31m0.183987868s) [upload=459258]: Got next segment. Inline: false Dec 27 11:00:00.312993763 (523h31m0.184076798s) [upload=459258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 11:00:00.313002319 (523h31m0.184085364s) [upload=459258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 11:00:00.552615684 (523h31m0.423698739s) [seg_pos={0 0}, upload=459258]: Begin upload segment... Dec 27 11:00:00.552671831 (523h31m0.423754876s) [seg_pos={0 0}, upload=459258]: Done begin upload segment. Dec 27 11:00:00.552709132 (523h31m0.423792177s) [noise=true , address=77.90.31.15:4700 , piece_id=RQ7NU7SBPN3VBQKP232D2VWUTQDAGBHC52MQAQEOLF3734H5GMXQ, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.552832787 (523h31m0.423915832s) [noise=true , address=45.91.116.11:4120 , piece_id=IO3SDPXLYVMHPDOD4NL2DS57A2GSQPDT5ASSDBEOCOJXLOV7GUJA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.553533782 (523h31m0.424616827s) [noise=true , address=77.90.31.19:4841 , piece_id=S2AVTWF6FGI6SYNXVQ5NERGCW5SB476ZNFCIRHEJILLBQNXPHE3A, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.553615759 (523h31m0.424698804s) [noise=true , address=145.79.227.39:4015 , piece_id=PILMKOXTMSTN7R5RQM7IHL2APILKUEXXMHANLFLKJQMWP5DKLC4A, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.554113897 (523h31m0.425196932s) [noise=true , address=145.79.227.41:4086 , piece_id=6P6FHECZXOAX7Y22WT2SXEO4YSQOLXCWKHNXU5TKZSVHJY6G34LA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.554536421 (523h31m0.425619456s) [noise=true , address=145.79.227.4:4186 , piece_id=P5KC4HEHONGB3LF5AMP2IJWFISUVZCYBGZU2AEJRVW2W7S2QTWGA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.554961851 (523h31m0.426044896s) [noise=true , address=145.79.227.13:4802 , piece_id=62J35OTB76CBOEOMYEW7HZSOJXSDKDAULNL5R2F3Y3JPH5J673GA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.555303622 (523h31m0.426386657s) [noise=true , address=45.91.116.13:4156 , piece_id=IWVAB3U63NFNI3OCSZZ2SQAUFWJYB7QQRRR4BIIQJTX3ZUT3GU5Q, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.555586921 (523h31m0.426669967s) [noise=true , address=45.91.116.4:4937 , piece_id=UUMRSCX66UGWA5IM3NTJXRJADBVFUGFSEKZFRZYDBRIVU32MPUDQ, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.555877275 (523h31m0.42696031s) [noise=true , address=145.79.227.29:4943 , piece_id=ICMFUYSW3ZYP3SZVWNOHX6AJBH6HPF5RHAGGJHKRWHKIG2FRRHGQ, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.558922343 (523h31m0.430005378s) [noise=true , address=45.91.116.6:4976 , piece_id=J36CLDRR3HHCHKCFDW2EEPGUNCHGDLJDBALCUHM5KGHZIDIEC6MA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.559323187 (523h31m0.430406232s) [noise=true , address=145.79.227.48:4302 , piece_id=GL74EFCSIM6FSUS4QYMTTECSGSUA5L6LQU53OT5JVC56NDFFFIAA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.559729069 (523h31m0.430812114s) [noise=true , address=145.79.227.42:4121 , piece_id=HAAPXEPF646CALJF3P4IX4QFMSSRGQSOD2NMM6AS4V2ALSCN2EOA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.560116456 (523h31m0.431199501s) [noise=true , address=145.79.227.37:4567 , piece_id=WYKEDPPE6DL46CS5KFXCWAGSLGPIANF3WP7ANN63L5ZZQLP4OCBA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.560495709 (523h31m0.431578744s) [noise=true , address=145.79.227.3:4149 , piece_id=S6SDXTBIR7ZVGBPISQONARGM4HWDWA2KX2ZLIE25EHNO4OECC3UA, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:00.560870202 (523h31m0.431953247s) [noise=true , address=145.79.227.17:4972 , piece_id=KHP6SC5OMJX5Q5NW7OGBZKWLSW4TRZXZCFBNHMONQIIDLJ3JDO6A, seg_pos={0 0}, upload=459258]: Uploading piece... Dec 27 11:00:02.244788575 (523h31m2.11587162s) [upload=459258]: Next returned nil segment Dec 27 11:00:02.244808653 (523h31m2.115891698s) [upload=459258]: Waiting for error group managing segments... Dec 27 11:00:02.692155475 (523h31m2.56323852s) [noise=true , address=145.79.227.13:4802 , piece_id=62J35OTB76CBOEOMYEW7HZSOJXSDKDAULNL5R2F3Y3JPH5J673GA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.703934054 (523h31m2.575017099s) [noise=true , address=145.79.227.17:4972 , piece_id=KHP6SC5OMJX5Q5NW7OGBZKWLSW4TRZXZCFBNHMONQIIDLJ3JDO6A, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.704255526 (523h31m2.575338571s) [noise=true , address=145.79.227.42:4121 , piece_id=HAAPXEPF646CALJF3P4IX4QFMSSRGQSOD2NMM6AS4V2ALSCN2EOA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.711942484 (523h31m2.583025529s) [noise=true , address=145.79.227.37:4567 , piece_id=WYKEDPPE6DL46CS5KFXCWAGSLGPIANF3WP7ANN63L5ZZQLP4OCBA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.712866783 (523h31m2.583949828s) [noise=true , address=145.79.227.4:4186 , piece_id=P5KC4HEHONGB3LF5AMP2IJWFISUVZCYBGZU2AEJRVW2W7S2QTWGA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.716436651 (523h31m2.587519706s) [noise=true , address=145.79.227.39:4015 , piece_id=PILMKOXTMSTN7R5RQM7IHL2APILKUEXXMHANLFLKJQMWP5DKLC4A, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.726893953 (523h31m2.597976998s) [noise=true , address=145.79.227.29:4943 , piece_id=ICMFUYSW3ZYP3SZVWNOHX6AJBH6HPF5RHAGGJHKRWHKIG2FRRHGQ, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.728120539 (523h31m2.599203594s) [noise=true , address=145.79.227.3:4149 , piece_id=S6SDXTBIR7ZVGBPISQONARGM4HWDWA2KX2ZLIE25EHNO4OECC3UA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.732899660 (523h31m2.603982705s) [noise=true , address=145.79.227.41:4086 , piece_id=6P6FHECZXOAX7Y22WT2SXEO4YSQOLXCWKHNXU5TKZSVHJY6G34LA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.733659487 (523h31m2.604742532s) [noise=true , address=145.79.227.48:4302 , piece_id=GL74EFCSIM6FSUS4QYMTTECSGSUA5L6LQU53OT5JVC56NDFFFIAA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.872994394 (523h31m2.744077449s) [noise=true , address=77.90.31.15:4700 , piece_id=RQ7NU7SBPN3VBQKP232D2VWUTQDAGBHC52MQAQEOLF3734H5GMXQ, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.919477646 (523h31m2.790560701s) [noise=true , address=45.91.116.13:4156 , piece_id=IWVAB3U63NFNI3OCSZZ2SQAUFWJYB7QQRRR4BIIQJTX3ZUT3GU5Q, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.937031331 (523h31m2.808114376s) [noise=true , address=45.91.116.4:4937 , piece_id=UUMRSCX66UGWA5IM3NTJXRJADBVFUGFSEKZFRZYDBRIVU32MPUDQ, seg_pos={0 0}, upload=459258]: Done uploading piece. err: <nil> Dec 27 11:00:02.937055968 (523h31m2.808139013s) [seg_pos={0 0}, upload=459258]: Segment reached optimal threshold of 13 pieces. Dec 27 11:00:02.937292188 (523h31m2.808375233s) [noise=true , address=45.91.116.11:4120 , piece_id=IO3SDPXLYVMHPDOD4NL2DS57A2GSQPDT5ASSDBEOCOJXLOV7GUJA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 11:00:02.937300864 (523h31m2.808383909s) [noise=true , address=45.91.116.11:4120 , piece_id=IO3SDPXLYVMHPDOD4NL2DS57A2GSQPDT5ASSDBEOCOJXLOV7GUJA, seg_pos={0 0}, upload=459258]: Long tail cancellation due to optimal threshold Dec 27 11:00:02.937610213 (523h31m2.808693258s) [noise=true , address=45.91.116.6:4976 , piece_id=J36CLDRR3HHCHKCFDW2EEPGUNCHGDLJDBALCUHM5KGHZIDIEC6MA, seg_pos={0 0}, upload=459258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 11:00:02.937617677 (523h31m2.808700722s) [noise=true , address=45.91.116.6:4976 , piece_id=J36CLDRR3HHCHKCFDW2EEPGUNCHGDLJDBALCUHM5KGHZIDIEC6MA, seg_pos={0 0}, upload=459258]: Long tail cancellation due to optimal threshold Dec 27 11:00:02.938024562 (523h31m2.809107637s) [noise=true , address=77.90.31.19:4841 , piece_id=S2AVTWF6FGI6SYNXVQ5NERGCW5SB476ZNFCIRHEJILLBQNXPHE3A, seg_pos={0 0}, upload=459258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:44276->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:44276->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44276->77.90.31.19:4841: use of closed network connection Dec 27 11:00:02.938034210 (523h31m2.809117256s) [noise=true , address=77.90.31.19:4841 , piece_id=S2AVTWF6FGI6SYNXVQ5NERGCW5SB476ZNFCIRHEJILLBQNXPHE3A, seg_pos={0 0}, upload=459258]: Long tail cancellation due to optimal threshold Dec 27 11:00:02.938058707 (523h31m2.809141752s) [seg_pos={0 0}, upload=459258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 11:00:02.938156052 (523h31m2.809239097s) [upload=459258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 11:00:02.938163958 (523h31m2.809247003s) [upload=459258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 11:00:03.243147488 (523h31m3.114230543s) [upload=459258]: Done uploading segments... Dec 27 11:00:03.243177295 (523h31m3.11426034s) [upload=459258]: Upload finished. err: <nil> Duration: 30.002633198s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.XX] duration: 30.002897788s trace: https://jaeger.dev.storj.io/trace/27054a31f2b8d23e FAIL (2025-12-27 10:59:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7ABFUIS4KRZBHBK742U4CBS4HKDQR6WNRM3GCO3SGGBOGIMGQPVA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BZTAB6TNAX5RESZP3RV4JSQYDQTWVFBMZNZAGXP2LHGSCV3L43VA) context deadline exceeded ======LOGS====== Dec 27 10:59:30.067062641 (523h30m29.938145676s) [upload=459248]: Starting upload Dec 27 10:59:30.067174163 (523h30m29.938257199s) [upload=459248]: Done starting upload Dec 27 10:59:30.067241672 (523h30m29.938324707s) [upload=459248]: Uploading segments... Dec 27 10:59:30.067360568 (523h30m29.938443603s) [upload=459248]: Got next segment. Inline: false Dec 27 10:59:30.067396206 (523h30m29.938479251s) [upload=459248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:59:30.067401556 (523h30m29.938484601s) [upload=459248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:59:30.311887309 (523h30m30.182970374s) [seg_pos={0 0}, upload=459248]: Begin upload segment... Dec 27 10:59:30.311940710 (523h30m30.183023755s) [seg_pos={0 0}, upload=459248]: Done begin upload segment. Dec 27 10:59:30.311978953 (523h30m30.183061988s) [noise=true , address=45.91.116.5:4299 , piece_id=7ABFUIS4KRZBHBK742U4CBS4HKDQR6WNRM3GCO3SGGBOGIMGQPVA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.312004281 (523h30m30.183087316s) [noise=true , address=45.91.116.10:4082 , piece_id=Y5HQQFOFGGQO5HRRD54GG3XIJE54NO7NAPNZSN2A37WFBAJTOUSA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.312389094 (523h30m30.183472149s) [noise=true , address=145.79.227.36:4517 , piece_id=IR52GMY3MNHTYK3ZYBJKJMTJ6GTZGD7UZCLVT4FYXCLS6WMQ7XRQ, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.312808262 (523h30m30.183891317s) [noise=true , address=145.79.227.46:4601 , piece_id=JPVAK5YZLACJGSSRY2HHBLCPYPLIOXXRMHJ5BR5MUGN5VITTVPZA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.313156495 (523h30m30.18423953s) [noise=true , address=145.79.227.37:4566 , piece_id=ZHDK6L7WMDBA47PXOGGIWW5Q6YIH7QZTHNSC4B6CPILCJM2AMVZQ, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.313412082 (523h30m30.184495117s) [noise=true , address=145.79.227.14:4837 , piece_id=VDURZDEWZVX52HKMRAQLTDUBETRUZOQBHXBCRHZ4MDADJA3UENUA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.313684871 (523h30m30.184767916s) [noise=true , address=145.79.227.7:4294 , piece_id=Z3BKQG3XCHOQEG3K2O7CMKVTQ6VWQ24AEZKVGSWR43N3NJ5DLK5A, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.314147883 (523h30m30.185230928s) [noise=true , address=145.79.227.49:4338 , piece_id=SLFLUWQ77PZ2TK4DUGEFTMND723GVXZSHOU2FPJMYJFTSOU2COTA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.314182449 (523h30m30.185265484s) [noise=true , address=45.91.116.7:4010 , piece_id=BZTAB6TNAX5RESZP3RV4JSQYDQTWVFBMZNZAGXP2LHGSCV3L43VA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.314538266 (523h30m30.185621312s) [noise=true , address=145.79.227.45:4229 , piece_id=QWAENQTQ4RCIX3YG35J44LKKQR5MGVXIWA7FMDHTAWRGDW3WO3SA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.314669326 (523h30m30.185752371s) [noise=true , address=145.79.227.11:4730 , piece_id=ZX5FPQ6AYRJ3LKSSHAJFNFCBEZBMDGB6MBRTOROAF7DCVYSEIKUA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.314909955 (523h30m30.185993s) [noise=true , address=145.79.227.34:4907 , piece_id=RZBH6RBKERLG2HIFO52YPGI7PUI3BLB42B2MXUQGJTQSGKWPZHMA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.315127288 (523h30m30.186210333s) [noise=true , address=77.90.31.10:4520 , piece_id=VMCG264FH6XVQ2WFSZSOGLGHW5KHMI7JZRLRQ63DVKCRVFYGPZGA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.315638121 (523h30m30.186721166s) [noise=true , address=45.91.116.9:4261 , piece_id=NJUH453L47D55KY5CP7SPKPHAEN6NAAJ2YMGHMSJ4KW3WSI32YAQ, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.317078183 (523h30m30.188161218s) [noise=true , address=145.79.227.41:4087 , piece_id=WLK5DZGN5EIV2IOFUVOEXFZDSOSHM37EZZRY5LL7DG7DS472BJXA, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:30.318079209 (523h30m30.189162254s) [noise=true , address=77.90.31.15:4697 , piece_id=GFJ4LH3TZRSX3ITJREERXNFBBVFRCXG756JNY7ZXJQMXFP27HEVQ, seg_pos={0 0}, upload=459248]: Uploading piece... Dec 27 10:59:32.024985115 (523h30m31.89606815s) [upload=459248]: Next returned nil segment Dec 27 10:59:32.025006937 (523h30m31.896089972s) [upload=459248]: Waiting for error group managing segments... Dec 27 10:59:32.306089725 (523h30m32.17717276s) [noise=true , address=145.79.227.49:4338 , piece_id=SLFLUWQ77PZ2TK4DUGEFTMND723GVXZSHOU2FPJMYJFTSOU2COTA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.320914174 (523h30m32.191997209s) [noise=true , address=145.79.227.46:4601 , piece_id=JPVAK5YZLACJGSSRY2HHBLCPYPLIOXXRMHJ5BR5MUGN5VITTVPZA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.324099379 (523h30m32.195182414s) [noise=true , address=145.79.227.7:4294 , piece_id=Z3BKQG3XCHOQEG3K2O7CMKVTQ6VWQ24AEZKVGSWR43N3NJ5DLK5A, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.324903419 (523h30m32.195986455s) [noise=true , address=145.79.227.37:4566 , piece_id=ZHDK6L7WMDBA47PXOGGIWW5Q6YIH7QZTHNSC4B6CPILCJM2AMVZQ, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.325548909 (523h30m32.196631944s) [noise=true , address=145.79.227.14:4837 , piece_id=VDURZDEWZVX52HKMRAQLTDUBETRUZOQBHXBCRHZ4MDADJA3UENUA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.329471950 (523h30m32.200554995s) [noise=true , address=145.79.227.11:4730 , piece_id=ZX5FPQ6AYRJ3LKSSHAJFNFCBEZBMDGB6MBRTOROAF7DCVYSEIKUA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.330677925 (523h30m32.20176097s) [noise=true , address=145.79.227.36:4517 , piece_id=IR52GMY3MNHTYK3ZYBJKJMTJ6GTZGD7UZCLVT4FYXCLS6WMQ7XRQ, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.334949219 (523h30m32.206032254s) [noise=true , address=145.79.227.41:4087 , piece_id=WLK5DZGN5EIV2IOFUVOEXFZDSOSHM37EZZRY5LL7DG7DS472BJXA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.345302874 (523h30m32.216385909s) [noise=true , address=145.79.227.45:4229 , piece_id=QWAENQTQ4RCIX3YG35J44LKKQR5MGVXIWA7FMDHTAWRGDW3WO3SA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.510718923 (523h30m32.381801958s) [noise=true , address=77.90.31.10:4520 , piece_id=VMCG264FH6XVQ2WFSZSOGLGHW5KHMI7JZRLRQ63DVKCRVFYGPZGA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.555870710 (523h30m32.426953755s) [noise=true , address=45.91.116.10:4082 , piece_id=Y5HQQFOFGGQO5HRRD54GG3XIJE54NO7NAPNZSN2A37WFBAJTOUSA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.591244768 (523h30m32.462327813s) [noise=true , address=45.91.116.7:4010 , piece_id=BZTAB6TNAX5RESZP3RV4JSQYDQTWVFBMZNZAGXP2LHGSCV3L43VA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.691923538 (523h30m32.563006583s) [noise=true , address=45.91.116.5:4299 , piece_id=7ABFUIS4KRZBHBK742U4CBS4HKDQR6WNRM3GCO3SGGBOGIMGQPVA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: <nil> Dec 27 10:59:32.691942063 (523h30m32.563025098s) [seg_pos={0 0}, upload=459248]: Segment reached optimal threshold of 13 pieces. Dec 27 10:59:32.692118318 (523h30m32.563201353s) [noise=true , address=145.79.227.34:4907 , piece_id=RZBH6RBKERLG2HIFO52YPGI7PUI3BLB42B2MXUQGJTQSGKWPZHMA, seg_pos={0 0}, upload=459248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 10:59:32.692125543 (523h30m32.563208578s) [noise=true , address=145.79.227.34:4907 , piece_id=RZBH6RBKERLG2HIFO52YPGI7PUI3BLB42B2MXUQGJTQSGKWPZHMA, seg_pos={0 0}, upload=459248]: Long tail cancellation due to optimal threshold Dec 27 10:59:32.692314933 (523h30m32.563397968s) [noise=true , address=77.90.31.15:4697 , piece_id=GFJ4LH3TZRSX3ITJREERXNFBBVFRCXG756JNY7ZXJQMXFP27HEVQ, seg_pos={0 0}, upload=459248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 10:59:32.692320774 (523h30m32.563403809s) [noise=true , address=77.90.31.15:4697 , piece_id=GFJ4LH3TZRSX3ITJREERXNFBBVFRCXG756JNY7ZXJQMXFP27HEVQ, seg_pos={0 0}, upload=459248]: Long tail cancellation due to optimal threshold Dec 27 10:59:32.692562314 (523h30m32.563645349s) [noise=true , address=45.91.116.9:4261 , piece_id=NJUH453L47D55KY5CP7SPKPHAEN6NAAJ2YMGHMSJ4KW3WSI32YAQ, seg_pos={0 0}, upload=459248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:43780->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 27 10:59:32.692578195 (523h30m32.56366123s) [noise=true , address=45.91.116.9:4261 , piece_id=NJUH453L47D55KY5CP7SPKPHAEN6NAAJ2YMGHMSJ4KW3WSI32YAQ, seg_pos={0 0}, upload=459248]: Long tail cancellation due to optimal threshold Dec 27 10:59:32.692614814 (523h30m32.563697849s) [seg_pos={0 0}, upload=459248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:59:32.692719584 (523h30m32.563802619s) [upload=459248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:59:32.692726357 (523h30m32.563809392s) [upload=459248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:59:32.986022341 (523h30m32.857105376s) [upload=459248]: Done uploading segments... Dec 27 10:59:32.986057127 (523h30m32.857140163s) [upload=459248]: Upload finished. err: <nil> Duration: 30.002922565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.X] duration: 30.002897796s trace: https://jaeger.dev.storj.io/trace/605ea23412a8a463 FAIL (2025-12-27 10:59:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:58:32.608586909 (523h29m32.479669954s) [upload=459235]: Starting upload Dec 27 10:58:32.608663775 (523h29m32.47974682s) [upload=459235]: Done starting upload Dec 27 10:58:32.608749258 (523h29m32.479832314s) [upload=459235]: Uploading segments... Dec 27 10:58:32.609020966 (523h29m32.480104021s) [upload=459235]: Got next segment. Inline: false Dec 27 10:58:32.609066362 (523h29m32.480149407s) [upload=459235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:58:32.609072504 (523h29m32.480155549s) [upload=459235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:58:32.852066896 (523h29m32.723149942s) [seg_pos={0 0}, upload=459235]: Begin upload segment... Dec 27 10:58:32.852128423 (523h29m32.723211468s) [seg_pos={0 0}, upload=459235]: Done begin upload segment. Dec 27 10:58:32.852211101 (523h29m32.723294136s) [noise=true , address=145.79.227.46:4602 , piece_id=JK6P3HSRHK6WPZKJRGWDBNGYYWDTEJJVCS3W4ILG73OKUBVKXOHQ, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.852236189 (523h29m32.723319234s) [noise=true , address=45.91.116.8:4045 , piece_id=L4JB7VT7A27ALAY5IKVJIXEUIBWCROGCPIU5MBQL47TLUKGXNTTA, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.854313243 (523h29m32.725396288s) [noise=true , address=45.91.116.12:4225 , piece_id=UBJRAZ4MJIKO77UFJ7CUEFPLDANCW2K44DCPNEPL32S3NCJP7USQ, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.856059098 (523h29m32.727142143s) [noise=true , address=145.79.227.36:4519 , piece_id=MCUQ45QEXGWTEWKQKXXQ6S35PFZDF6DCWOKMYGJPOEDTLBEQIFLQ, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.857869154 (523h29m32.728952189s) [noise=true , address=145.79.227.40:4050 , piece_id=KLENUSKFIJYFV7QFW4UN4WAMT3EIZQQZNLZYPTT6JKTVNHUV57YA, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.862044896 (523h29m32.733127941s) [noise=true , address=145.79.227.16:4910 , piece_id=RI53OALVKS7PJCJZUI5OJAFIJMWX2DUB2VJXJOTNDLYRHLAKBW5Q, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.863574999 (523h29m32.734658034s) [noise=true , address=145.79.227.42:4121 , piece_id=TAAN3ZBACXTHFBG6UEAGW5FTC3MIYHSGY3XGYD6UERSKBLQLLYIQ, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.866578670 (523h29m32.737661705s) [noise=true , address=145.79.227.8:4331 , piece_id=VZXZRIKVRVA77XTSBWDEMGZWTMW3CFTWCGJQMXSFXWO74D675LCA, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.869151940 (523h29m32.740234975s) [noise=true , address=45.91.116.6:4974 , piece_id=ZL2PL7PF5GRHVSRLAHFKB4EUQ3NF2A7KU22S7LJR64MAAYVZTQWA, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.871588569 (523h29m32.742671604s) [noise=true , address=145.79.227.45:4230 , piece_id=GNHJ6KXFW6HNGEXO7YCFO4ZNSTBDMHQOKPJUWS63J5LJJDVG4RCA, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.873930619 (523h29m32.745013654s) [noise=true , address=145.79.227.11:4732 , piece_id=MMMFNJERRWN5QAUR5S3YW3KYTYOU42DDP33WNMLZCT4YAS6R2CPQ, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.875501700 (523h29m32.746584745s) [noise=true , address=45.91.116.14:4191 , piece_id=7IF2XHFUSAXMSMCKYZRHMLW4IRKSX326GDAD7I6METKOHFHOTT6A, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.877188241 (523h29m32.748271276s) [noise=true , address=45.91.116.4:4940 , piece_id=UBNNBTP2GRXV2HG4DBTHJUWGOSIECPZGJGGVLIQUCKNV2GWKTJZQ, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.877522839 (523h29m32.748605884s) [noise=true , address=45.91.116.5:4300 , piece_id=EUDCUWTJ2P7SXBEMDHEIPATS7L4BR7SPKWFU4QUPJFLOLYOL3HLA, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.879694514 (523h29m32.750777549s) [noise=true , address=145.79.227.39:4013 , piece_id=B2KXZADBPME6RWESLNKL5KXQSMU5F32GHYYNAWPQPPHYDN5XUD6A, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:32.880940226 (523h29m32.752023271s) [noise=true , address=45.91.116.7:4009 , piece_id=ZDQIPMZOKPYPGPLS7VD5VETVDHXC3VRBD35ARWJNEKHZF6MQN22A, seg_pos={0 0}, upload=459235]: Uploading piece... Dec 27 10:58:34.586244290 (523h29m34.457327335s) [upload=459235]: Next returned nil segment Dec 27 10:58:34.586258618 (523h29m34.457341653s) [upload=459235]: Waiting for error group managing segments... Dec 27 10:58:34.872927274 (523h29m34.744010319s) [noise=true , address=145.79.227.36:4519 , piece_id=MCUQ45QEXGWTEWKQKXXQ6S35PFZDF6DCWOKMYGJPOEDTLBEQIFLQ, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:34.876824054 (523h29m34.747907089s) [noise=true , address=145.79.227.45:4230 , piece_id=GNHJ6KXFW6HNGEXO7YCFO4ZNSTBDMHQOKPJUWS63J5LJJDVG4RCA, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:34.886260673 (523h29m34.757343708s) [noise=true , address=145.79.227.39:4013 , piece_id=B2KXZADBPME6RWESLNKL5KXQSMU5F32GHYYNAWPQPPHYDN5XUD6A, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:34.889531080 (523h29m34.760614115s) [noise=true , address=145.79.227.46:4602 , piece_id=JK6P3HSRHK6WPZKJRGWDBNGYYWDTEJJVCS3W4ILG73OKUBVKXOHQ, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:34.892015911 (523h29m34.763098946s) [noise=true , address=145.79.227.16:4910 , piece_id=RI53OALVKS7PJCJZUI5OJAFIJMWX2DUB2VJXJOTNDLYRHLAKBW5Q, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:34.900195647 (523h29m34.771278682s) [noise=true , address=145.79.227.42:4121 , piece_id=TAAN3ZBACXTHFBG6UEAGW5FTC3MIYHSGY3XGYD6UERSKBLQLLYIQ, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.124723386 (523h29m34.995806431s) [noise=true , address=45.91.116.12:4225 , piece_id=UBJRAZ4MJIKO77UFJ7CUEFPLDANCW2K44DCPNEPL32S3NCJP7USQ, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.141672119 (523h29m35.012755154s) [noise=true , address=45.91.116.5:4300 , piece_id=EUDCUWTJ2P7SXBEMDHEIPATS7L4BR7SPKWFU4QUPJFLOLYOL3HLA, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.189270104 (523h29m35.060353139s) [noise=true , address=45.91.116.4:4940 , piece_id=UBNNBTP2GRXV2HG4DBTHJUWGOSIECPZGJGGVLIQUCKNV2GWKTJZQ, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.196129786 (523h29m35.067212821s) [noise=true , address=45.91.116.8:4045 , piece_id=L4JB7VT7A27ALAY5IKVJIXEUIBWCROGCPIU5MBQL47TLUKGXNTTA, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.252398531 (523h29m35.123481566s) [noise=true , address=45.91.116.6:4974 , piece_id=ZL2PL7PF5GRHVSRLAHFKB4EUQ3NF2A7KU22S7LJR64MAAYVZTQWA, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.262396679 (523h29m35.133479714s) [noise=true , address=45.91.116.7:4009 , piece_id=ZDQIPMZOKPYPGPLS7VD5VETVDHXC3VRBD35ARWJNEKHZF6MQN22A, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.724979785 (523h29m35.59606283s) [noise=true , address=145.79.227.11:4732 , piece_id=MMMFNJERRWN5QAUR5S3YW3KYTYOU42DDP33WNMLZCT4YAS6R2CPQ, seg_pos={0 0}, upload=459235]: Done uploading piece. err: <nil> Dec 27 10:58:35.725000774 (523h29m35.59608381s) [seg_pos={0 0}, upload=459235]: Segment reached optimal threshold of 13 pieces. Dec 27 10:58:35.725256081 (523h29m35.596339126s) [noise=true , address=145.79.227.8:4331 , piece_id=VZXZRIKVRVA77XTSBWDEMGZWTMW3CFTWCGJQMXSFXWO74D675LCA, seg_pos={0 0}, upload=459235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: expected piece hash; context canceled; EOF Dec 27 10:58:35.725283694 (523h29m35.596366729s) [noise=true , address=145.79.227.8:4331 , piece_id=VZXZRIKVRVA77XTSBWDEMGZWTMW3CFTWCGJQMXSFXWO74D675LCA, seg_pos={0 0}, upload=459235]: Long tail cancellation due to optimal threshold Dec 27 10:58:35.725385317 (523h29m35.596468352s) [noise=true , address=145.79.227.40:4050 , piece_id=KLENUSKFIJYFV7QFW4UN4WAMT3EIZQQZNLZYPTT6JKTVNHUV57YA, seg_pos={0 0}, upload=459235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 10:58:35.725403662 (523h29m35.596486697s) [noise=true , address=145.79.227.40:4050 , piece_id=KLENUSKFIJYFV7QFW4UN4WAMT3EIZQQZNLZYPTT6JKTVNHUV57YA, seg_pos={0 0}, upload=459235]: Long tail cancellation due to optimal threshold Dec 27 10:58:35.725521336 (523h29m35.596604391s) [noise=true , address=45.91.116.14:4191 , piece_id=7IF2XHFUSAXMSMCKYZRHMLW4IRKSX326GDAD7I6METKOHFHOTT6A, seg_pos={0 0}, upload=459235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 10:58:35.725531336 (523h29m35.596614391s) [noise=true , address=45.91.116.14:4191 , piece_id=7IF2XHFUSAXMSMCKYZRHMLW4IRKSX326GDAD7I6METKOHFHOTT6A, seg_pos={0 0}, upload=459235]: Long tail cancellation due to optimal threshold Dec 27 10:58:35.725565360 (523h29m35.596648395s) [seg_pos={0 0}, upload=459235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:58:35.725645473 (523h29m35.596728518s) [upload=459235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:58:35.725651243 (523h29m35.596734288s) [upload=459235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:58:36.036787279 (523h29m35.907870314s) [upload=459235]: Done uploading segments... Dec 27 10:58:36.036805053 (523h29m35.907888098s) [upload=459235]: Upload finished. err: <nil> Duration: 30.002923524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX.] duration: 6.144298084s trace: https://jaeger.dev.storj.io/trace/76c4531117b3b827 FAIL (2025-12-27 10:57:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:57:22.356539116 (523h28m22.227622151s) [upload=459218]: Starting upload Dec 27 10:57:22.356600503 (523h28m22.227683548s) [upload=459218]: Done starting upload Dec 27 10:57:22.356787819 (523h28m22.227870854s) [upload=459218]: Uploading segments... Dec 27 10:57:22.356823948 (523h28m22.227906993s) [upload=459218]: Got next segment. Inline: false Dec 27 10:57:22.356863192 (523h28m22.227946237s) [upload=459218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:57:22.356868082 (523h28m22.227951127s) [upload=459218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:57:22.602849161 (523h28m22.473932206s) [seg_pos={0 0}, upload=459218]: Begin upload segment... Dec 27 10:57:22.602903644 (523h28m22.473986689s) [seg_pos={0 0}, upload=459218]: Done begin upload segment. Dec 27 10:57:22.602934863 (523h28m22.474017899s) [noise=true , address=145.79.227.45:4230 , piece_id=XODX4U5FOY4NP7PMUQAFSVGQ77LJCQJQC44ZPN7QAICIXLAHUCCQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.603673900 (523h28m22.474756946s) [noise=true , address=45.91.116.9:4263 , piece_id=BXBYTB2EQPD2BSFZS2YKQOSVYXBDMY6GES7SH665CXD2MMZTUSAA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.604332725 (523h28m22.47541577s) [noise=true , address=145.79.227.49:4337 , piece_id=3YAWSDTXSC4VEECXESYC5FDIC7YVB4VQ4YKVMD7NPACB5CYERREQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.604837617 (523h28m22.475920662s) [noise=true , address=45.91.116.14:4192 , piece_id=WFSBXUALZ7KOD2TSXGHNFYZZQZLT4KLNTYGKGUMR5MBME54ECPLA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.605276823 (523h28m22.476359858s) [noise=true , address=145.79.227.39:4014 , piece_id=ETULVYHRDADRPSJLRHEF3TGNXY3XNQG36NHITP3OTQM5POAILX2A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.606441080 (523h28m22.477524125s) [noise=true , address=45.91.116.10:4083 , piece_id=SQRQJZ64EY3YXF2FANJN44BLT24DV34LB3PFINCSWBUBUSUWELBA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607036253 (523h28m22.478119298s) [noise=true , address=145.79.227.40:4050 , piece_id=B2PB5U3BASXWSOAVQPLJV3Z4DWOZZ5FOCQDO6VAZDGQMRLNTIG5A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607280709 (523h28m22.478363754s) [noise=true , address=145.79.227.42:4123 , piece_id=EFDPVTNSNGRZH7C7W6PGAKI7K73EJ2LGZCXA5BAIP3BZPNROWN2A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607459499 (523h28m22.478542544s) [noise=true , address=45.91.116.12:4225 , piece_id=DNWUZERVIOOUELK4L6YQAM23EHQXZPCH5VGGNRGNDOCQ7W6GMAXA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607979750 (523h28m22.479062785s) [noise=true , address=77.90.31.14:4662 , piece_id=HPLH34N4PDITTPYJQXE6NG3OQKOKCWO3MCJLT4S7R2A2XPVYUEUA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.608039424 (523h28m22.479122459s) [noise=true , address=145.79.227.50:4375 , piece_id=UGHW6W326HPJGT6G7QTDJ4WNYD6M63NM52Z7BV563ZYBJ6QETHWQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.608694831 (523h28m22.479777896s) [noise=true , address=145.79.227.16:4910 , piece_id=R3MB53AFFC5D32IMTKRYBYIV7T52STVLA6KS6O4SNBZDLOO2HTXA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.609416315 (523h28m22.48049935s) [noise=true , address=45.91.116.11:4120 , piece_id=XXKSSJB4VPMVI64PQNSGDVYCXZZIOZH55DEVZV23OHO77I2PGU2A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.610209656 (523h28m22.481292701s) [noise=true , address=145.79.227.46:4602 , piece_id=NG24Y7JW2SJLU3ZCSXZIBVRLG7IUVMVQC2XP5NR3MRSUYG6YE52A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.611957835 (523h28m22.48304088s) [noise=true , address=145.79.227.38:4979 , piece_id=VORNYYPXN7FQZZ2YMVBDJU4UG53WOGVPS2BG3ISPWOENFQNF6T5A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.613343393 (523h28m22.484426438s) [noise=true , address=145.79.227.29:4943 , piece_id=QGTPB3B5ZN4TQLR2PWFWCFGY32RSVXSJZWKOCLFZUYUXA5S2ZZKQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:24.312361850 (523h28m24.183444885s) [upload=459218]: Next returned nil segment Dec 27 10:57:24.312379344 (523h28m24.183462389s) [upload=459218]: Waiting for error group managing segments... Dec 27 10:57:24.601647778 (523h28m24.472730823s) [noise=true , address=145.79.227.40:4050 , piece_id=B2PB5U3BASXWSOAVQPLJV3Z4DWOZZ5FOCQDO6VAZDGQMRLNTIG5A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.606613124 (523h28m24.477696169s) [noise=true , address=145.79.227.42:4123 , piece_id=EFDPVTNSNGRZH7C7W6PGAKI7K73EJ2LGZCXA5BAIP3BZPNROWN2A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.611659493 (523h28m24.482742528s) [noise=true , address=145.79.227.45:4230 , piece_id=XODX4U5FOY4NP7PMUQAFSVGQ77LJCQJQC44ZPN7QAICIXLAHUCCQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.612435751 (523h28m24.483518786s) [noise=true , address=145.79.227.49:4337 , piece_id=3YAWSDTXSC4VEECXESYC5FDIC7YVB4VQ4YKVMD7NPACB5CYERREQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.617611767 (523h28m24.488694812s) [noise=true , address=145.79.227.39:4014 , piece_id=ETULVYHRDADRPSJLRHEF3TGNXY3XNQG36NHITP3OTQM5POAILX2A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.628487015 (523h28m24.49957006s) [noise=true , address=145.79.227.29:4943 , piece_id=QGTPB3B5ZN4TQLR2PWFWCFGY32RSVXSJZWKOCLFZUYUXA5S2ZZKQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.628637001 (523h28m24.499720046s) [noise=true , address=145.79.227.16:4910 , piece_id=R3MB53AFFC5D32IMTKRYBYIV7T52STVLA6KS6O4SNBZDLOO2HTXA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.638843335 (523h28m24.50992637s) [noise=true , address=145.79.227.46:4602 , piece_id=NG24Y7JW2SJLU3ZCSXZIBVRLG7IUVMVQC2XP5NR3MRSUYG6YE52A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.640020015 (523h28m24.51110306s) [noise=true , address=145.79.227.38:4979 , piece_id=VORNYYPXN7FQZZ2YMVBDJU4UG53WOGVPS2BG3ISPWOENFQNF6T5A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.660204901 (523h28m24.531287936s) [noise=true , address=145.79.227.50:4375 , piece_id=UGHW6W326HPJGT6G7QTDJ4WNYD6M63NM52Z7BV563ZYBJ6QETHWQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.873591625 (523h28m24.74467466s) [noise=true , address=45.91.116.9:4263 , piece_id=BXBYTB2EQPD2BSFZS2YKQOSVYXBDMY6GES7SH665CXD2MMZTUSAA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.971210118 (523h28m24.842293173s) [noise=true , address=45.91.116.10:4083 , piece_id=SQRQJZ64EY3YXF2FANJN44BLT24DV34LB3PFINCSWBUBUSUWELBA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:25.047644843 (523h28m24.918727888s) [noise=true , address=45.91.116.12:4225 , piece_id=DNWUZERVIOOUELK4L6YQAM23EHQXZPCH5VGGNRGNDOCQ7W6GMAXA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:25.047669019 (523h28m24.918752054s) [seg_pos={0 0}, upload=459218]: Segment reached optimal threshold of 13 pieces. Dec 27 10:57:25.047855895 (523h28m24.91893894s) [noise=true , address=45.91.116.14:4192 , piece_id=WFSBXUALZ7KOD2TSXGHNFYZZQZLT4KLNTYGKGUMR5MBME54ECPLA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 10:57:25.047862166 (523h28m24.918945211s) [noise=true , address=45.91.116.14:4192 , piece_id=WFSBXUALZ7KOD2TSXGHNFYZZQZLT4KLNTYGKGUMR5MBME54ECPLA, seg_pos={0 0}, upload=459218]: Long tail cancellation due to optimal threshold Dec 27 10:57:25.047972677 (523h28m24.919055712s) [noise=true , address=45.91.116.11:4120 , piece_id=XXKSSJB4VPMVI64PQNSGDVYCXZZIOZH55DEVZV23OHO77I2PGU2A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:42664->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:42664->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42664->45.91.116.11:4120: use of closed network connection Dec 27 10:57:25.047978067 (523h28m24.919061102s) [noise=true , address=45.91.116.11:4120 , piece_id=XXKSSJB4VPMVI64PQNSGDVYCXZZIOZH55DEVZV23OHO77I2PGU2A, seg_pos={0 0}, upload=459218]: Long tail cancellation due to optimal threshold Dec 27 10:57:25.048074340 (523h28m24.919157395s) [noise=true , address=77.90.31.14:4662 , piece_id=HPLH34N4PDITTPYJQXE6NG3OQKOKCWO3MCJLT4S7R2A2XPVYUEUA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:57:25.048112793 (523h28m24.919195838s) [noise=true , address=77.90.31.14:4662 , piece_id=HPLH34N4PDITTPYJQXE6NG3OQKOKCWO3MCJLT4S7R2A2XPVYUEUA, seg_pos={0 0}, upload=459218]: Long tail cancellation due to optimal threshold Dec 27 10:57:25.048207764 (523h28m24.91929081s) [seg_pos={0 0}, upload=459218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:57:25.048421541 (523h28m24.919504596s) [upload=459218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:57:25.048429937 (523h28m24.919512982s) [upload=459218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:57:25.335672385 (523h28m25.20675542s) [upload=459218]: Done uploading segments... Dec 27 10:57:25.335693145 (523h28m25.20677618s) [upload=459218]: Upload finished. err: <nil> Duration: 6.144315338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.XX] duration: 30.003111639s trace: https://jaeger.dev.storj.io/trace/76c4531117b3b827 FAIL (2025-12-27 10:57:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:57:22.356539116 (523h28m22.227622151s) [upload=459218]: Starting upload Dec 27 10:57:22.356600503 (523h28m22.227683548s) [upload=459218]: Done starting upload Dec 27 10:57:22.356787819 (523h28m22.227870854s) [upload=459218]: Uploading segments... Dec 27 10:57:22.356823948 (523h28m22.227906993s) [upload=459218]: Got next segment. Inline: false Dec 27 10:57:22.356863192 (523h28m22.227946237s) [upload=459218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:57:22.356868082 (523h28m22.227951127s) [upload=459218]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:57:22.602849161 (523h28m22.473932206s) [seg_pos={0 0}, upload=459218]: Begin upload segment... Dec 27 10:57:22.602903644 (523h28m22.473986689s) [seg_pos={0 0}, upload=459218]: Done begin upload segment. Dec 27 10:57:22.602934863 (523h28m22.474017899s) [noise=true , address=145.79.227.45:4230 , piece_id=XODX4U5FOY4NP7PMUQAFSVGQ77LJCQJQC44ZPN7QAICIXLAHUCCQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.603673900 (523h28m22.474756946s) [noise=true , address=45.91.116.9:4263 , piece_id=BXBYTB2EQPD2BSFZS2YKQOSVYXBDMY6GES7SH665CXD2MMZTUSAA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.604332725 (523h28m22.47541577s) [noise=true , address=145.79.227.49:4337 , piece_id=3YAWSDTXSC4VEECXESYC5FDIC7YVB4VQ4YKVMD7NPACB5CYERREQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.604837617 (523h28m22.475920662s) [noise=true , address=45.91.116.14:4192 , piece_id=WFSBXUALZ7KOD2TSXGHNFYZZQZLT4KLNTYGKGUMR5MBME54ECPLA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.605276823 (523h28m22.476359858s) [noise=true , address=145.79.227.39:4014 , piece_id=ETULVYHRDADRPSJLRHEF3TGNXY3XNQG36NHITP3OTQM5POAILX2A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.606441080 (523h28m22.477524125s) [noise=true , address=45.91.116.10:4083 , piece_id=SQRQJZ64EY3YXF2FANJN44BLT24DV34LB3PFINCSWBUBUSUWELBA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607036253 (523h28m22.478119298s) [noise=true , address=145.79.227.40:4050 , piece_id=B2PB5U3BASXWSOAVQPLJV3Z4DWOZZ5FOCQDO6VAZDGQMRLNTIG5A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607280709 (523h28m22.478363754s) [noise=true , address=145.79.227.42:4123 , piece_id=EFDPVTNSNGRZH7C7W6PGAKI7K73EJ2LGZCXA5BAIP3BZPNROWN2A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607459499 (523h28m22.478542544s) [noise=true , address=45.91.116.12:4225 , piece_id=DNWUZERVIOOUELK4L6YQAM23EHQXZPCH5VGGNRGNDOCQ7W6GMAXA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.607979750 (523h28m22.479062785s) [noise=true , address=77.90.31.14:4662 , piece_id=HPLH34N4PDITTPYJQXE6NG3OQKOKCWO3MCJLT4S7R2A2XPVYUEUA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.608039424 (523h28m22.479122459s) [noise=true , address=145.79.227.50:4375 , piece_id=UGHW6W326HPJGT6G7QTDJ4WNYD6M63NM52Z7BV563ZYBJ6QETHWQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.608694831 (523h28m22.479777896s) [noise=true , address=145.79.227.16:4910 , piece_id=R3MB53AFFC5D32IMTKRYBYIV7T52STVLA6KS6O4SNBZDLOO2HTXA, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.609416315 (523h28m22.48049935s) [noise=true , address=45.91.116.11:4120 , piece_id=XXKSSJB4VPMVI64PQNSGDVYCXZZIOZH55DEVZV23OHO77I2PGU2A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.610209656 (523h28m22.481292701s) [noise=true , address=145.79.227.46:4602 , piece_id=NG24Y7JW2SJLU3ZCSXZIBVRLG7IUVMVQC2XP5NR3MRSUYG6YE52A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.611957835 (523h28m22.48304088s) [noise=true , address=145.79.227.38:4979 , piece_id=VORNYYPXN7FQZZ2YMVBDJU4UG53WOGVPS2BG3ISPWOENFQNF6T5A, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:22.613343393 (523h28m22.484426438s) [noise=true , address=145.79.227.29:4943 , piece_id=QGTPB3B5ZN4TQLR2PWFWCFGY32RSVXSJZWKOCLFZUYUXA5S2ZZKQ, seg_pos={0 0}, upload=459218]: Uploading piece... Dec 27 10:57:24.312361850 (523h28m24.183444885s) [upload=459218]: Next returned nil segment Dec 27 10:57:24.312379344 (523h28m24.183462389s) [upload=459218]: Waiting for error group managing segments... Dec 27 10:57:24.601647778 (523h28m24.472730823s) [noise=true , address=145.79.227.40:4050 , piece_id=B2PB5U3BASXWSOAVQPLJV3Z4DWOZZ5FOCQDO6VAZDGQMRLNTIG5A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.606613124 (523h28m24.477696169s) [noise=true , address=145.79.227.42:4123 , piece_id=EFDPVTNSNGRZH7C7W6PGAKI7K73EJ2LGZCXA5BAIP3BZPNROWN2A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.611659493 (523h28m24.482742528s) [noise=true , address=145.79.227.45:4230 , piece_id=XODX4U5FOY4NP7PMUQAFSVGQ77LJCQJQC44ZPN7QAICIXLAHUCCQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.612435751 (523h28m24.483518786s) [noise=true , address=145.79.227.49:4337 , piece_id=3YAWSDTXSC4VEECXESYC5FDIC7YVB4VQ4YKVMD7NPACB5CYERREQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.617611767 (523h28m24.488694812s) [noise=true , address=145.79.227.39:4014 , piece_id=ETULVYHRDADRPSJLRHEF3TGNXY3XNQG36NHITP3OTQM5POAILX2A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.628487015 (523h28m24.49957006s) [noise=true , address=145.79.227.29:4943 , piece_id=QGTPB3B5ZN4TQLR2PWFWCFGY32RSVXSJZWKOCLFZUYUXA5S2ZZKQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.628637001 (523h28m24.499720046s) [noise=true , address=145.79.227.16:4910 , piece_id=R3MB53AFFC5D32IMTKRYBYIV7T52STVLA6KS6O4SNBZDLOO2HTXA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.638843335 (523h28m24.50992637s) [noise=true , address=145.79.227.46:4602 , piece_id=NG24Y7JW2SJLU3ZCSXZIBVRLG7IUVMVQC2XP5NR3MRSUYG6YE52A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.640020015 (523h28m24.51110306s) [noise=true , address=145.79.227.38:4979 , piece_id=VORNYYPXN7FQZZ2YMVBDJU4UG53WOGVPS2BG3ISPWOENFQNF6T5A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.660204901 (523h28m24.531287936s) [noise=true , address=145.79.227.50:4375 , piece_id=UGHW6W326HPJGT6G7QTDJ4WNYD6M63NM52Z7BV563ZYBJ6QETHWQ, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.873591625 (523h28m24.74467466s) [noise=true , address=45.91.116.9:4263 , piece_id=BXBYTB2EQPD2BSFZS2YKQOSVYXBDMY6GES7SH665CXD2MMZTUSAA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:24.971210118 (523h28m24.842293173s) [noise=true , address=45.91.116.10:4083 , piece_id=SQRQJZ64EY3YXF2FANJN44BLT24DV34LB3PFINCSWBUBUSUWELBA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:25.047644843 (523h28m24.918727888s) [noise=true , address=45.91.116.12:4225 , piece_id=DNWUZERVIOOUELK4L6YQAM23EHQXZPCH5VGGNRGNDOCQ7W6GMAXA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: <nil> Dec 27 10:57:25.047669019 (523h28m24.918752054s) [seg_pos={0 0}, upload=459218]: Segment reached optimal threshold of 13 pieces. Dec 27 10:57:25.047855895 (523h28m24.91893894s) [noise=true , address=45.91.116.14:4192 , piece_id=WFSBXUALZ7KOD2TSXGHNFYZZQZLT4KLNTYGKGUMR5MBME54ECPLA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 10:57:25.047862166 (523h28m24.918945211s) [noise=true , address=45.91.116.14:4192 , piece_id=WFSBXUALZ7KOD2TSXGHNFYZZQZLT4KLNTYGKGUMR5MBME54ECPLA, seg_pos={0 0}, upload=459218]: Long tail cancellation due to optimal threshold Dec 27 10:57:25.047972677 (523h28m24.919055712s) [noise=true , address=45.91.116.11:4120 , piece_id=XXKSSJB4VPMVI64PQNSGDVYCXZZIOZH55DEVZV23OHO77I2PGU2A, seg_pos={0 0}, upload=459218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:42664->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:42664->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42664->45.91.116.11:4120: use of closed network connection Dec 27 10:57:25.047978067 (523h28m24.919061102s) [noise=true , address=45.91.116.11:4120 , piece_id=XXKSSJB4VPMVI64PQNSGDVYCXZZIOZH55DEVZV23OHO77I2PGU2A, seg_pos={0 0}, upload=459218]: Long tail cancellation due to optimal threshold Dec 27 10:57:25.048074340 (523h28m24.919157395s) [noise=true , address=77.90.31.14:4662 , piece_id=HPLH34N4PDITTPYJQXE6NG3OQKOKCWO3MCJLT4S7R2A2XPVYUEUA, seg_pos={0 0}, upload=459218]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:57:25.048112793 (523h28m24.919195838s) [noise=true , address=77.90.31.14:4662 , piece_id=HPLH34N4PDITTPYJQXE6NG3OQKOKCWO3MCJLT4S7R2A2XPVYUEUA, seg_pos={0 0}, upload=459218]: Long tail cancellation due to optimal threshold Dec 27 10:57:25.048207764 (523h28m24.91929081s) [seg_pos={0 0}, upload=459218]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:57:25.048421541 (523h28m24.919504596s) [upload=459218]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:57:25.048429937 (523h28m24.919512982s) [upload=459218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:57:25.335672385 (523h28m25.20675542s) [upload=459218]: Done uploading segments... Dec 27 10:57:25.335693145 (523h28m25.20677618s) [upload=459218]: Upload finished. err: <nil> Duration: 30.00313347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.X] duration: 30.00265732s trace: https://jaeger.dev.storj.io/trace/6934be372504460a FAIL (2025-12-27 10:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:56:36.865264528 (523h27m36.736347573s) [upload=459205]: Starting upload Dec 27 10:56:36.865344551 (523h27m36.736427596s) [upload=459205]: Done starting upload Dec 27 10:56:36.865391721 (523h27m36.736474766s) [upload=459205]: Uploading segments... Dec 27 10:56:36.865701130 (523h27m36.736784175s) [upload=459205]: Got next segment. Inline: false Dec 27 10:56:36.865738221 (523h27m36.736821256s) [upload=459205]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:56:36.865743490 (523h27m36.736826526s) [upload=459205]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:56:37.110416760 (523h27m36.981499805s) [seg_pos={0 0}, upload=459205]: Begin upload segment... Dec 27 10:56:37.110476013 (523h27m36.981559058s) [seg_pos={0 0}, upload=459205]: Done begin upload segment. Dec 27 10:56:37.110510067 (523h27m36.981593102s) [noise=true , address=145.79.227.38:4978 , piece_id=KVOGZGCWWB6K3Y3MHW7UKBMYG74DNHAGDEHI5TVJ2VHWDKOUHR2Q, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.111625211 (523h27m36.982708246s) [noise=true , address=145.79.227.30:4761 , piece_id=K5YM46KF653SO7YULOJA2ESDFIEN2FZYKNPOOOVHIFHHNCGSIMJA, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.111765709 (523h27m36.982848744s) [noise=true , address=145.79.227.4:4185 , piece_id=KS3VI77SGFKA53HVYUEPZZE4RPI2UDDBRXBY3TOBJTHHZWKIWNLQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.112160811 (523h27m36.983243856s) [noise=true , address=145.79.227.17:4971 , piece_id=GLWDMF6TQPGVEB4YFTNPSJT26FJJNFICPPEFAUZAI62B5ECPCACQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.112193343 (523h27m36.983276388s) [noise=true , address=145.79.227.51:4411 , piece_id=VNKHZL23W33NBEDKLQYVXGBEV6IZTKTYNBBDLHE3C77K57IYY7TA, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.112576903 (523h27m36.983659948s) [noise=true , address=45.91.116.11:4119 , piece_id=BYU5ORH6WUZXRRO7AIXTK3QTNB4PNUMP2JCA5AMQPQTIZ3G5JGLA, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.112670762 (523h27m36.983753807s) [noise=true , address=45.91.116.14:4189 , piece_id=TYWBW2HLYHF2DCWNXE2BSBEJ7WHJJXXTWNW3X3355CSC73N2RQPQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.112903405 (523h27m36.98398644s) [noise=true , address=45.91.116.6:4975 , piece_id=ELKQRVJOYASIMMP7SJ6D7TWSGDU4Y6D4NK7JA5B6Y6GANDTHB7HQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.113066846 (523h27m36.984149891s) [noise=true , address=45.91.116.9:4263 , piece_id=Q5HYZRY7HNAPS2OUL6ATUKSLLNHUTXKSKTPW3WYRGQNC7QGS5TQA, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.113291233 (523h27m36.984374268s) [noise=true , address=145.79.227.39:4014 , piece_id=Y2LDERM57LY6UIGAX5LNMGATIQ5JUSIW7HNPQPSKSR5OV6UHMAYQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.113531561 (523h27m36.984614596s) [noise=true , address=145.79.227.50:4374 , piece_id=2DVZIO2PYIABCZOPSAO7WJW7AU4P3XQ276ZYIUAXWOONJGHLMXYQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.113707065 (523h27m36.9847901s) [noise=true , address=45.91.116.7:4012 , piece_id=XCTHM4Y6MESX7WOPEQWWV3OLVUBFEAXSJFW3QKZDDQC4NEMZ5URA, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.114031894 (523h27m36.985114939s) [noise=true , address=145.79.227.47:4267 , piece_id=VEBKDKD4XVRHVRHKK5I2NM7KS4KAHKPZPGTKWCB33JDDEZ77QNRQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.114481059 (523h27m36.985564104s) [noise=true , address=45.91.116.4:4940 , piece_id=D7P5PURHKC2O44IB2FJXVADV2RQVFXZEYUEKHWDOAHBELAVCBQAA, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.114906259 (523h27m36.985989314s) [noise=true , address=45.91.116.12:4227 , piece_id=ZUUSTYLFGK3SGWKSS4B5W3NZUCXRLT5RDZHS46RE5UGO3ZS56SFA, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:37.115316861 (523h27m36.986399896s) [noise=true , address=77.90.31.18:4805 , piece_id=HLG2KDLGE27CCVUZFBQB2XZO7R3D5ZKF7PXQ6DQCEQXVSBFEAHKQ, seg_pos={0 0}, upload=459205]: Uploading piece... Dec 27 10:56:38.719097188 (523h27m38.590180223s) [upload=459205]: Next returned nil segment Dec 27 10:56:38.719108700 (523h27m38.590191735s) [upload=459205]: Waiting for error group managing segments... Dec 27 10:56:39.119583121 (523h27m38.990666166s) [noise=true , address=145.79.227.4:4185 , piece_id=KS3VI77SGFKA53HVYUEPZZE4RPI2UDDBRXBY3TOBJTHHZWKIWNLQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.120517800 (523h27m38.991600835s) [noise=true , address=145.79.227.50:4374 , piece_id=2DVZIO2PYIABCZOPSAO7WJW7AU4P3XQ276ZYIUAXWOONJGHLMXYQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.121452680 (523h27m38.992535715s) [noise=true , address=145.79.227.51:4411 , piece_id=VNKHZL23W33NBEDKLQYVXGBEV6IZTKTYNBBDLHE3C77K57IYY7TA, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.131155456 (523h27m39.002238491s) [noise=true , address=145.79.227.38:4978 , piece_id=KVOGZGCWWB6K3Y3MHW7UKBMYG74DNHAGDEHI5TVJ2VHWDKOUHR2Q, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.143801586 (523h27m39.014884621s) [noise=true , address=145.79.227.39:4014 , piece_id=Y2LDERM57LY6UIGAX5LNMGATIQ5JUSIW7HNPQPSKSR5OV6UHMAYQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.155475214 (523h27m39.026558249s) [noise=true , address=145.79.227.47:4267 , piece_id=VEBKDKD4XVRHVRHKK5I2NM7KS4KAHKPZPGTKWCB33JDDEZ77QNRQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.164850878 (523h27m39.035933913s) [noise=true , address=145.79.227.17:4971 , piece_id=GLWDMF6TQPGVEB4YFTNPSJT26FJJNFICPPEFAUZAI62B5ECPCACQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.300905799 (523h27m39.171988834s) [noise=true , address=77.90.31.18:4805 , piece_id=HLG2KDLGE27CCVUZFBQB2XZO7R3D5ZKF7PXQ6DQCEQXVSBFEAHKQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.373994002 (523h27m39.245077037s) [noise=true , address=45.91.116.4:4940 , piece_id=D7P5PURHKC2O44IB2FJXVADV2RQVFXZEYUEKHWDOAHBELAVCBQAA, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.386502389 (523h27m39.257585434s) [noise=true , address=45.91.116.7:4012 , piece_id=XCTHM4Y6MESX7WOPEQWWV3OLVUBFEAXSJFW3QKZDDQC4NEMZ5URA, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.402509460 (523h27m39.273592505s) [noise=true , address=45.91.116.12:4227 , piece_id=ZUUSTYLFGK3SGWKSS4B5W3NZUCXRLT5RDZHS46RE5UGO3ZS56SFA, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.478252126 (523h27m39.349335161s) [noise=true , address=45.91.116.11:4119 , piece_id=BYU5ORH6WUZXRRO7AIXTK3QTNB4PNUMP2JCA5AMQPQTIZ3G5JGLA, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.482373965 (523h27m39.353457s) [noise=true , address=45.91.116.9:4263 , piece_id=Q5HYZRY7HNAPS2OUL6ATUKSLLNHUTXKSKTPW3WYRGQNC7QGS5TQA, seg_pos={0 0}, upload=459205]: Done uploading piece. err: <nil> Dec 27 10:56:39.482394434 (523h27m39.353477469s) [seg_pos={0 0}, upload=459205]: Segment reached optimal threshold of 13 pieces. Dec 27 10:56:39.482624913 (523h27m39.353707948s) [noise=true , address=145.79.227.30:4761 , piece_id=K5YM46KF653SO7YULOJA2ESDFIEN2FZYKNPOOOVHIFHHNCGSIMJA, seg_pos={0 0}, upload=459205]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 10:56:39.482633229 (523h27m39.353716264s) [noise=true , address=145.79.227.30:4761 , piece_id=K5YM46KF653SO7YULOJA2ESDFIEN2FZYKNPOOOVHIFHHNCGSIMJA, seg_pos={0 0}, upload=459205]: Long tail cancellation due to optimal threshold Dec 27 10:56:39.482653457 (523h27m39.353736493s) [noise=true , address=45.91.116.14:4189 , piece_id=TYWBW2HLYHF2DCWNXE2BSBEJ7WHJJXXTWNW3X3355CSC73N2RQPQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 10:56:39.482661152 (523h27m39.353744187s) [noise=true , address=45.91.116.14:4189 , piece_id=TYWBW2HLYHF2DCWNXE2BSBEJ7WHJJXXTWNW3X3355CSC73N2RQPQ, seg_pos={0 0}, upload=459205]: Long tail cancellation due to optimal threshold Dec 27 10:56:39.482760391 (523h27m39.353843426s) [noise=true , address=45.91.116.6:4975 , piece_id=ELKQRVJOYASIMMP7SJ6D7TWSGDU4Y6D4NK7JA5B6Y6GANDTHB7HQ, seg_pos={0 0}, upload=459205]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 10:56:39.482765951 (523h27m39.353848987s) [noise=true , address=45.91.116.6:4975 , piece_id=ELKQRVJOYASIMMP7SJ6D7TWSGDU4Y6D4NK7JA5B6Y6GANDTHB7HQ, seg_pos={0 0}, upload=459205]: Long tail cancellation due to optimal threshold Dec 27 10:56:39.482786301 (523h27m39.353869336s) [seg_pos={0 0}, upload=459205]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:56:39.482860422 (523h27m39.353943457s) [upload=459205]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:56:39.482867765 (523h27m39.3539508s) [upload=459205]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:56:39.768791231 (523h27m39.639874297s) [upload=459205]: Done uploading segments... Dec 27 10:56:39.768820267 (523h27m39.639903302s) [upload=459205]: Upload finished. err: <nil> Duration: 30.002677879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X.] duration: 6.029405141s trace: https://jaeger.dev.storj.io/trace/6210427fe8b23bad FAIL (2025-12-27 10:56:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: RYIKDKM357TRR6WTH4LHA6334B3LMBTZJ27XDAE54JRMS37INKCQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: J3EYN2UH7PADEWRU2AJWUDRUZO6IPNQ66CNYPIYAMOH5MAHJOMHA) context deadline exceeded ======LOGS====== Dec 27 10:55:02.648263703 (523h26m2.519346758s) [upload=459183]: Starting upload Dec 27 10:55:02.648446220 (523h26m2.519529265s) [upload=459183]: Done starting upload Dec 27 10:55:02.648537264 (523h26m2.519620319s) [upload=459183]: Uploading segments... Dec 27 10:55:02.649512831 (523h26m2.520595916s) [upload=459183]: Got next segment. Inline: false Dec 27 10:55:02.649573225 (523h26m2.520656271s) [upload=459183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:55:02.649581181 (523h26m2.520664226s) [upload=459183]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:55:02.898846192 (523h26m2.769929247s) [seg_pos={0 0}, upload=459183]: Begin upload segment... Dec 27 10:55:02.899033971 (523h26m2.770117016s) [seg_pos={0 0}, upload=459183]: Done begin upload segment. Dec 27 10:55:02.899085709 (523h26m2.770168784s) [noise=true , address=145.79.227.37:4565 , piece_id=772DA52JKYPYG2U7KCXXSBKZXNSLCAROIJ534XXEIDRP6HQCC4BQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.899129872 (523h26m2.770212918s) [noise=true , address=145.79.227.14:4837 , piece_id=PWHVBQYGHSYN6HS7NBKZPJTPZWEAMTGLETENO5AFNP3R4XG5HSKA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.899225094 (523h26m2.770308149s) [noise=true , address=145.79.227.23:4546 , piece_id=RIDIS2HMAL4FHBSG42WVWXHVRR5PWNM7NZSLQ66V3IP5AW7OJ3PQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.899803155 (523h26m2.7708862s) [noise=true , address=145.79.227.41:4086 , piece_id=4IUJYIJXGDVLOOMVE4OE66XSGAOMYS5WHBCFHDUV2I5CN7E7HO3A, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.902020978 (523h26m2.773104023s) [noise=true , address=145.79.227.51:4411 , piece_id=N326YIA3JV7AO3EY3YLWSL4H2MCLO5ODYPT3QZ6FDFTQSPFDQW7A, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.902739135 (523h26m2.77382218s) [noise=true , address=77.90.31.18:4805 , piece_id=DCEIZMFU57KQWPMYIM5KDDJF24537FIKFE27SJ4SXVC6PQVXNSTQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.903317297 (523h26m2.774400352s) [noise=true , address=145.79.227.16:4911 , piece_id=IGUVKDHFTGV6DAHTMVSGU3NJHRNUQZHHUAGN7YJHM33W7BOYBDJA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.905357060 (523h26m2.776440105s) [noise=true , address=145.79.227.7:4295 , piece_id=4GUE3TJR6F45BN63HIBD73GBQ2IKJNTKY5HT7VAQMXG4UNQ2Y3KQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.905605223 (523h26m2.776688278s) [noise=true , address=145.79.227.48:4303 , piece_id=YCPTUDNV3M5RBYIEQP35TJZTZWO5GNPMTKISX2KZOYPQN7HLSJEA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.907551739 (523h26m2.778634784s) [noise=true , address=77.90.31.19:4844 , piece_id=27KG4RV6UZA5EALGKWIEPJDEHNULI5LZDQA73C4E6XQBUSXSMS3A, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.908697281 (523h26m2.779780326s) [noise=true , address=77.90.31.15:4700 , piece_id=RYIKDKM357TRR6WTH4LHA6334B3LMBTZJ27XDAE54JRMS37INKCQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.910632897 (523h26m2.781715942s) [noise=true , address=145.79.227.47:4266 , piece_id=QRNLYQKTCEONBGGNGMB74BQGNAWFNLURVN6NE62TN3BKWYZYYEQA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.912918720 (523h26m2.784001765s) [noise=true , address=45.91.116.5:4297 , piece_id=5P5WAEM6ZTHTLNQMI6MRHGUWCUXNFDJYWBOJEOZA3JMGFUUHJUTQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.915231623 (523h26m2.786314668s) [noise=true , address=77.90.31.10:4517 , piece_id=J3EYN2UH7PADEWRU2AJWUDRUZO6IPNQ66CNYPIYAMOH5MAHJOMHA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.921010618 (523h26m2.792093663s) [noise=true , address=145.79.227.17:4970 , piece_id=RKDVXNRTKQYLKSKBP5DIB2MX7HZVFVG5QHRWQ73BNTI5MZEHWWRA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.925116155 (523h26m2.79619923s) [noise=true , address=145.79.227.39:4013 , piece_id=XH63HTNWRNTIZGMCL2N3FPFVCK564FSIFH4DMRSLMVWJRWYXROHA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:04.534578933 (523h26m4.405661978s) [upload=459183]: Next returned nil segment Dec 27 10:55:04.534595754 (523h26m4.405678799s) [upload=459183]: Waiting for error group managing segments... Dec 27 10:55:04.925512826 (523h26m4.796595871s) [noise=true , address=145.79.227.14:4837 , piece_id=PWHVBQYGHSYN6HS7NBKZPJTPZWEAMTGLETENO5AFNP3R4XG5HSKA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.927919018 (523h26m4.799002063s) [noise=true , address=145.79.227.23:4546 , piece_id=RIDIS2HMAL4FHBSG42WVWXHVRR5PWNM7NZSLQ66V3IP5AW7OJ3PQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.940663736 (523h26m4.811746781s) [noise=true , address=145.79.227.41:4086 , piece_id=4IUJYIJXGDVLOOMVE4OE66XSGAOMYS5WHBCFHDUV2I5CN7E7HO3A, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.951381033 (523h26m4.822464078s) [noise=true , address=145.79.227.51:4411 , piece_id=N326YIA3JV7AO3EY3YLWSL4H2MCLO5ODYPT3QZ6FDFTQSPFDQW7A, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.953018792 (523h26m4.824101837s) [noise=true , address=145.79.227.48:4303 , piece_id=YCPTUDNV3M5RBYIEQP35TJZTZWO5GNPMTKISX2KZOYPQN7HLSJEA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.954938828 (523h26m4.826021883s) [noise=true , address=145.79.227.37:4565 , piece_id=772DA52JKYPYG2U7KCXXSBKZXNSLCAROIJ534XXEIDRP6HQCC4BQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.960241125 (523h26m4.83132417s) [noise=true , address=145.79.227.39:4013 , piece_id=XH63HTNWRNTIZGMCL2N3FPFVCK564FSIFH4DMRSLMVWJRWYXROHA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.966508369 (523h26m4.837591404s) [noise=true , address=145.79.227.47:4266 , piece_id=QRNLYQKTCEONBGGNGMB74BQGNAWFNLURVN6NE62TN3BKWYZYYEQA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.973641892 (523h26m4.844724948s) [noise=true , address=145.79.227.7:4295 , piece_id=4GUE3TJR6F45BN63HIBD73GBQ2IKJNTKY5HT7VAQMXG4UNQ2Y3KQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.976996029 (523h26m4.848079104s) [noise=true , address=145.79.227.17:4970 , piece_id=RKDVXNRTKQYLKSKBP5DIB2MX7HZVFVG5QHRWQ73BNTI5MZEHWWRA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.978750721 (523h26m4.849833766s) [noise=true , address=145.79.227.16:4911 , piece_id=IGUVKDHFTGV6DAHTMVSGU3NJHRNUQZHHUAGN7YJHM33W7BOYBDJA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:05.125246424 (523h26m4.996329469s) [noise=true , address=77.90.31.10:4517 , piece_id=J3EYN2UH7PADEWRU2AJWUDRUZO6IPNQ66CNYPIYAMOH5MAHJOMHA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:05.134392148 (523h26m5.005475193s) [noise=true , address=77.90.31.15:4700 , piece_id=RYIKDKM357TRR6WTH4LHA6334B3LMBTZJ27XDAE54JRMS37INKCQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:05.134418799 (523h26m5.005501854s) [seg_pos={0 0}, upload=459183]: Segment reached optimal threshold of 13 pieces. Dec 27 10:55:05.134714021 (523h26m5.005797076s) [noise=true , address=77.90.31.19:4844 , piece_id=27KG4RV6UZA5EALGKWIEPJDEHNULI5LZDQA73C4E6XQBUSXSMS3A, seg_pos={0 0}, upload=459183]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 10:55:05.134725732 (523h26m5.005808778s) [noise=true , address=77.90.31.19:4844 , piece_id=27KG4RV6UZA5EALGKWIEPJDEHNULI5LZDQA73C4E6XQBUSXSMS3A, seg_pos={0 0}, upload=459183]: Long tail cancellation due to optimal threshold Dec 27 10:55:05.134895476 (523h26m5.005978541s) [noise=true , address=45.91.116.5:4297 , piece_id=5P5WAEM6ZTHTLNQMI6MRHGUWCUXNFDJYWBOJEOZA3JMGFUUHJUTQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 27 10:55:05.134904162 (523h26m5.005987207s) [noise=true , address=45.91.116.5:4297 , piece_id=5P5WAEM6ZTHTLNQMI6MRHGUWCUXNFDJYWBOJEOZA3JMGFUUHJUTQ, seg_pos={0 0}, upload=459183]: Long tail cancellation due to optimal threshold Dec 27 10:55:05.135076300 (523h26m5.006159345s) [noise=true , address=77.90.31.18:4805 , piece_id=DCEIZMFU57KQWPMYIM5KDDJF24537FIKFE27SJ4SXVC6PQVXNSTQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 10:55:05.135085979 (523h26m5.006169024s) [noise=true , address=77.90.31.18:4805 , piece_id=DCEIZMFU57KQWPMYIM5KDDJF24537FIKFE27SJ4SXVC6PQVXNSTQ, seg_pos={0 0}, upload=459183]: Long tail cancellation due to optimal threshold Dec 27 10:55:05.135143989 (523h26m5.006227034s) [seg_pos={0 0}, upload=459183]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:55:05.135331537 (523h26m5.006414572s) [upload=459183]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:55:05.135339912 (523h26m5.006422947s) [upload=459183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:55:05.411931680 (523h26m5.283014765s) [upload=459183]: Done uploading segments... Dec 27 10:55:05.411962108 (523h26m5.283045183s) [upload=459183]: Upload finished. err: <nil> Duration: 6.029432642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........X] duration: 30.002308132s trace: https://jaeger.dev.storj.io/trace/6210427fe8b23bad FAIL (2025-12-27 10:55:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: RYIKDKM357TRR6WTH4LHA6334B3LMBTZJ27XDAE54JRMS37INKCQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: J3EYN2UH7PADEWRU2AJWUDRUZO6IPNQ66CNYPIYAMOH5MAHJOMHA) context deadline exceeded ======LOGS====== Dec 27 10:55:02.648263703 (523h26m2.519346758s) [upload=459183]: Starting upload Dec 27 10:55:02.648446220 (523h26m2.519529265s) [upload=459183]: Done starting upload Dec 27 10:55:02.648537264 (523h26m2.519620319s) [upload=459183]: Uploading segments... Dec 27 10:55:02.649512831 (523h26m2.520595916s) [upload=459183]: Got next segment. Inline: false Dec 27 10:55:02.649573225 (523h26m2.520656271s) [upload=459183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:55:02.649581181 (523h26m2.520664226s) [upload=459183]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:55:02.898846192 (523h26m2.769929247s) [seg_pos={0 0}, upload=459183]: Begin upload segment... Dec 27 10:55:02.899033971 (523h26m2.770117016s) [seg_pos={0 0}, upload=459183]: Done begin upload segment. Dec 27 10:55:02.899085709 (523h26m2.770168784s) [noise=true , address=145.79.227.37:4565 , piece_id=772DA52JKYPYG2U7KCXXSBKZXNSLCAROIJ534XXEIDRP6HQCC4BQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.899129872 (523h26m2.770212918s) [noise=true , address=145.79.227.14:4837 , piece_id=PWHVBQYGHSYN6HS7NBKZPJTPZWEAMTGLETENO5AFNP3R4XG5HSKA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.899225094 (523h26m2.770308149s) [noise=true , address=145.79.227.23:4546 , piece_id=RIDIS2HMAL4FHBSG42WVWXHVRR5PWNM7NZSLQ66V3IP5AW7OJ3PQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.899803155 (523h26m2.7708862s) [noise=true , address=145.79.227.41:4086 , piece_id=4IUJYIJXGDVLOOMVE4OE66XSGAOMYS5WHBCFHDUV2I5CN7E7HO3A, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.902020978 (523h26m2.773104023s) [noise=true , address=145.79.227.51:4411 , piece_id=N326YIA3JV7AO3EY3YLWSL4H2MCLO5ODYPT3QZ6FDFTQSPFDQW7A, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.902739135 (523h26m2.77382218s) [noise=true , address=77.90.31.18:4805 , piece_id=DCEIZMFU57KQWPMYIM5KDDJF24537FIKFE27SJ4SXVC6PQVXNSTQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.903317297 (523h26m2.774400352s) [noise=true , address=145.79.227.16:4911 , piece_id=IGUVKDHFTGV6DAHTMVSGU3NJHRNUQZHHUAGN7YJHM33W7BOYBDJA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.905357060 (523h26m2.776440105s) [noise=true , address=145.79.227.7:4295 , piece_id=4GUE3TJR6F45BN63HIBD73GBQ2IKJNTKY5HT7VAQMXG4UNQ2Y3KQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.905605223 (523h26m2.776688278s) [noise=true , address=145.79.227.48:4303 , piece_id=YCPTUDNV3M5RBYIEQP35TJZTZWO5GNPMTKISX2KZOYPQN7HLSJEA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.907551739 (523h26m2.778634784s) [noise=true , address=77.90.31.19:4844 , piece_id=27KG4RV6UZA5EALGKWIEPJDEHNULI5LZDQA73C4E6XQBUSXSMS3A, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.908697281 (523h26m2.779780326s) [noise=true , address=77.90.31.15:4700 , piece_id=RYIKDKM357TRR6WTH4LHA6334B3LMBTZJ27XDAE54JRMS37INKCQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.910632897 (523h26m2.781715942s) [noise=true , address=145.79.227.47:4266 , piece_id=QRNLYQKTCEONBGGNGMB74BQGNAWFNLURVN6NE62TN3BKWYZYYEQA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.912918720 (523h26m2.784001765s) [noise=true , address=45.91.116.5:4297 , piece_id=5P5WAEM6ZTHTLNQMI6MRHGUWCUXNFDJYWBOJEOZA3JMGFUUHJUTQ, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.915231623 (523h26m2.786314668s) [noise=true , address=77.90.31.10:4517 , piece_id=J3EYN2UH7PADEWRU2AJWUDRUZO6IPNQ66CNYPIYAMOH5MAHJOMHA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.921010618 (523h26m2.792093663s) [noise=true , address=145.79.227.17:4970 , piece_id=RKDVXNRTKQYLKSKBP5DIB2MX7HZVFVG5QHRWQ73BNTI5MZEHWWRA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:02.925116155 (523h26m2.79619923s) [noise=true , address=145.79.227.39:4013 , piece_id=XH63HTNWRNTIZGMCL2N3FPFVCK564FSIFH4DMRSLMVWJRWYXROHA, seg_pos={0 0}, upload=459183]: Uploading piece... Dec 27 10:55:04.534578933 (523h26m4.405661978s) [upload=459183]: Next returned nil segment Dec 27 10:55:04.534595754 (523h26m4.405678799s) [upload=459183]: Waiting for error group managing segments... Dec 27 10:55:04.925512826 (523h26m4.796595871s) [noise=true , address=145.79.227.14:4837 , piece_id=PWHVBQYGHSYN6HS7NBKZPJTPZWEAMTGLETENO5AFNP3R4XG5HSKA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.927919018 (523h26m4.799002063s) [noise=true , address=145.79.227.23:4546 , piece_id=RIDIS2HMAL4FHBSG42WVWXHVRR5PWNM7NZSLQ66V3IP5AW7OJ3PQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.940663736 (523h26m4.811746781s) [noise=true , address=145.79.227.41:4086 , piece_id=4IUJYIJXGDVLOOMVE4OE66XSGAOMYS5WHBCFHDUV2I5CN7E7HO3A, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.951381033 (523h26m4.822464078s) [noise=true , address=145.79.227.51:4411 , piece_id=N326YIA3JV7AO3EY3YLWSL4H2MCLO5ODYPT3QZ6FDFTQSPFDQW7A, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.953018792 (523h26m4.824101837s) [noise=true , address=145.79.227.48:4303 , piece_id=YCPTUDNV3M5RBYIEQP35TJZTZWO5GNPMTKISX2KZOYPQN7HLSJEA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.954938828 (523h26m4.826021883s) [noise=true , address=145.79.227.37:4565 , piece_id=772DA52JKYPYG2U7KCXXSBKZXNSLCAROIJ534XXEIDRP6HQCC4BQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.960241125 (523h26m4.83132417s) [noise=true , address=145.79.227.39:4013 , piece_id=XH63HTNWRNTIZGMCL2N3FPFVCK564FSIFH4DMRSLMVWJRWYXROHA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.966508369 (523h26m4.837591404s) [noise=true , address=145.79.227.47:4266 , piece_id=QRNLYQKTCEONBGGNGMB74BQGNAWFNLURVN6NE62TN3BKWYZYYEQA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.973641892 (523h26m4.844724948s) [noise=true , address=145.79.227.7:4295 , piece_id=4GUE3TJR6F45BN63HIBD73GBQ2IKJNTKY5HT7VAQMXG4UNQ2Y3KQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.976996029 (523h26m4.848079104s) [noise=true , address=145.79.227.17:4970 , piece_id=RKDVXNRTKQYLKSKBP5DIB2MX7HZVFVG5QHRWQ73BNTI5MZEHWWRA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:04.978750721 (523h26m4.849833766s) [noise=true , address=145.79.227.16:4911 , piece_id=IGUVKDHFTGV6DAHTMVSGU3NJHRNUQZHHUAGN7YJHM33W7BOYBDJA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:05.125246424 (523h26m4.996329469s) [noise=true , address=77.90.31.10:4517 , piece_id=J3EYN2UH7PADEWRU2AJWUDRUZO6IPNQ66CNYPIYAMOH5MAHJOMHA, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:05.134392148 (523h26m5.005475193s) [noise=true , address=77.90.31.15:4700 , piece_id=RYIKDKM357TRR6WTH4LHA6334B3LMBTZJ27XDAE54JRMS37INKCQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: <nil> Dec 27 10:55:05.134418799 (523h26m5.005501854s) [seg_pos={0 0}, upload=459183]: Segment reached optimal threshold of 13 pieces. Dec 27 10:55:05.134714021 (523h26m5.005797076s) [noise=true , address=77.90.31.19:4844 , piece_id=27KG4RV6UZA5EALGKWIEPJDEHNULI5LZDQA73C4E6XQBUSXSMS3A, seg_pos={0 0}, upload=459183]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 10:55:05.134725732 (523h26m5.005808778s) [noise=true , address=77.90.31.19:4844 , piece_id=27KG4RV6UZA5EALGKWIEPJDEHNULI5LZDQA73C4E6XQBUSXSMS3A, seg_pos={0 0}, upload=459183]: Long tail cancellation due to optimal threshold Dec 27 10:55:05.134895476 (523h26m5.005978541s) [noise=true , address=45.91.116.5:4297 , piece_id=5P5WAEM6ZTHTLNQMI6MRHGUWCUXNFDJYWBOJEOZA3JMGFUUHJUTQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 27 10:55:05.134904162 (523h26m5.005987207s) [noise=true , address=45.91.116.5:4297 , piece_id=5P5WAEM6ZTHTLNQMI6MRHGUWCUXNFDJYWBOJEOZA3JMGFUUHJUTQ, seg_pos={0 0}, upload=459183]: Long tail cancellation due to optimal threshold Dec 27 10:55:05.135076300 (523h26m5.006159345s) [noise=true , address=77.90.31.18:4805 , piece_id=DCEIZMFU57KQWPMYIM5KDDJF24537FIKFE27SJ4SXVC6PQVXNSTQ, seg_pos={0 0}, upload=459183]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 10:55:05.135085979 (523h26m5.006169024s) [noise=true , address=77.90.31.18:4805 , piece_id=DCEIZMFU57KQWPMYIM5KDDJF24537FIKFE27SJ4SXVC6PQVXNSTQ, seg_pos={0 0}, upload=459183]: Long tail cancellation due to optimal threshold Dec 27 10:55:05.135143989 (523h26m5.006227034s) [seg_pos={0 0}, upload=459183]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:55:05.135331537 (523h26m5.006414572s) [upload=459183]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:55:05.135339912 (523h26m5.006422947s) [upload=459183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:55:05.411931680 (523h26m5.283014765s) [upload=459183]: Done uploading segments... Dec 27 10:55:05.411962108 (523h26m5.283045183s) [upload=459183]: Upload finished. err: <nil> Duration: 30.002332159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX........] duration: 5.89908022s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:53:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 5.899123052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX.......] duration: 6.528098186s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 6.52812631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX......] duration: 5.685377804s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 5.685406418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX.....] duration: 5.849555296s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 5.84958367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX....] duration: 6.163681473s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:51:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 6.163724244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X..X.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX...] duration: 6.22271896s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 6.222745984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX..X.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX..] duration: 5.599775947s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:49:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 5.599801365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX..X.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX.] duration: 6.382185601s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:48:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 6.382213493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX..X.XXX.....XX...X..X....X..XX.XXX..XX...XX.X..X.XX.XXXX] duration: 30.002150827s trace: https://jaeger.dev.storj.io/trace/3e647f7825ca7fc0 FAIL (2025-12-27 10:48:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 10:47:43.809427592 (523h18m43.680510637s) [upload=459079]: Starting upload Dec 27 10:47:43.809623014 (523h18m43.680706129s) [upload=459079]: Done starting upload Dec 27 10:47:43.809724858 (523h18m43.680807903s) [upload=459079]: Uploading segments... Dec 27 10:47:43.810019138 (523h18m43.681102173s) [upload=459079]: Got next segment. Inline: false Dec 27 10:47:43.810059396 (523h18m43.681142431s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 10:47:43.810065156 (523h18m43.681148192s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 10:47:44.053507061 (523h18m43.924590106s) [seg_pos={0 0}, upload=459079]: Begin upload segment... Dec 27 10:47:44.053562747 (523h18m43.924645792s) [seg_pos={0 0}, upload=459079]: Done begin upload segment. Dec 27 10:47:44.053594468 (523h18m43.924677513s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.053669069 (523h18m43.924752105s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054559605 (523h18m43.92564265s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.054971309 (523h18m43.926054354s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055352795 (523h18m43.92643584s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.055726808 (523h18m43.926809853s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056120467 (523h18m43.927203512s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.056511521 (523h18m43.927594566s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.058562467 (523h18m43.929645512s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.059056278 (523h18m43.930139313s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.060985852 (523h18m43.932068887s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.062796309 (523h18m43.933879344s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.064246862 (523h18m43.935329907s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.066099269 (523h18m43.937182304s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067306588 (523h18m43.938389633s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:44.067598523 (523h18m43.938681558s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Uploading piece... Dec 27 10:47:45.767388419 (523h18m45.638471474s) [upload=459079]: Next returned nil segment Dec 27 10:47:45.767402355 (523h18m45.638485391s) [upload=459079]: Waiting for error group managing segments... Dec 27 10:47:46.056438239 (523h18m45.927521284s) [noise=true , address=145.79.227.14:4837 , piece_id=JYVWIXCML3QPD562YWNHMBNRHJXZPTI422AG27KVRMPCQSU7DXUQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.059131447 (523h18m45.930214492s) [noise=true , address=145.79.227.49:4338 , piece_id=7DFHB6XQLHVFTXLNXDYNYL5Q563IMMPKVZZUEQXSZJUYJ4JIZJUA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.072238725 (523h18m45.94332176s) [noise=true , address=145.79.227.36:4517 , piece_id=DEQIDZXX2A4DZ5AUGMXW5ZC5P6ZWJZOPDASTAW7L5EWXTJIBSISA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.073235894 (523h18m45.944318929s) [noise=true , address=145.79.227.51:4409 , piece_id=JXKJYMBSMF3XTYC3IACTI7N4ETIOTHT7DGOMSHRGUFRM5PFJFBGQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.076964734 (523h18m45.948047769s) [noise=true , address=145.79.227.3:4149 , piece_id=EVEHSMJWL5M7VIGJDKHBXPNO24SPW7RXEDT3UADVILJQC2RF2P2Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.077251069 (523h18m45.948334114s) [noise=true , address=145.79.227.47:4266 , piece_id=T6GRZUOQMYLQM3SDJOW5EOVCLJCH637IEYDHJVFCGBYUMXTYLXBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.085034270 (523h18m45.956117315s) [noise=true , address=145.79.227.48:4302 , piece_id=GYBGQCUISMHR2XJ3E44MSWWP7T4Q2I4B3DH3EU54UVJP4XSMI2BQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092210214 (523h18m45.963293259s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEW72RH44VOEWPXNTMO3KY6PCSYOG2A3WAYEUMCPMZRQW2VA7Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.092776743 (523h18m45.963859798s) [noise=true , address=145.79.227.37:4566 , piece_id=MMIUR3ZLAE22JGBHKFBOWI56KNAGF23XEQ5RGZDR75TNXF7UBQLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.097311557 (523h18m45.968394602s) [noise=true , address=145.79.227.29:4943 , piece_id=OE3OBWSRUARRX5YBBVVEUGYWZPWJEFNFFPDCSIBWJHGM4KFA776Q, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.476926969 (523h18m46.348010015s) [noise=true , address=45.91.116.6:4973 , piece_id=KA4EEPV5XBKF3Y2TU5WTKLLQZT6WJFL3ICAPRRDVMQVOUVNIQQBA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:46.723281911 (523h18m46.594364956s) [noise=true , address=45.91.116.12:4228 , piece_id=G3DATPT24DTFH5YNXCZWHO36A3ENZ66S3HPGG47TP2AEENRM3NGA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062733433 (523h18m46.933816478s) [noise=true , address=45.91.116.14:4192 , piece_id=56AHILA4NKTOOJK35NDBU7MJKHLR6UKWXAX3WLT4IIIISLN3EKLA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: <nil> Dec 27 10:47:47.062791223 (523h18m46.933874268s) [seg_pos={0 0}, upload=459079]: Segment reached optimal threshold of 13 pieces. Dec 27 10:47:47.063033795 (523h18m46.93411683s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063043062 (523h18m46.934126098s) [noise=true , address=45.91.116.9:4261 , piece_id=SFPTNA2KBOUWK75Q3E3NIY63PFD65MWECSZ7UMJVXHVMVUFF2BSQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063190765 (523h18m46.93427381s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063209319 (523h18m46.934292364s) [noise=true , address=77.90.31.14:4662 , piece_id=LH7FRFU7VQCSDZ2WMWQUWROE3SV3ZGLXWXWG3AROK3JP5RQIIKKQ, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063379303 (523h18m46.934462348s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 10:47:47.063388861 (523h18m46.934471896s) [noise=true , address=77.90.31.12:4589 , piece_id=2OMMML4QFKLKOZ2KNYCYGKE74DMTWKGOK3BDTZRIUBWSSYKOJTEA, seg_pos={0 0}, upload=459079]: Long tail cancellation due to optimal threshold Dec 27 10:47:47.063412637 (523h18m46.934495672s) [seg_pos={0 0}, upload=459079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 10:47:47.063503270 (523h18m46.934586305s) [upload=459079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 10:47:47.063510132 (523h18m46.934593188s) [upload=459079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 10:47:47.359679539 (523h18m47.230762574s) [upload=459079]: Done uploading segments... Dec 27 10:47:47.359696822 (523h18m47.230779857s) [upload=459079]: Upload finished. err: <nil> Duration: 30.002161847s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)