Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX........] duration: 7.945398452s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 7.945426856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..............X..............................] duration: 25.996833221s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 25.996877506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..............X.............................] duration: 5.950060283s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 5.950075251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.......] duration: 6.081986414s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:48:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.08200578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..............X............................] duration: 24.044554577s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:48:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 24.044576468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX......] duration: 8.880183509s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 8.880207595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.....] duration: 26.336873741s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:48:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 26.336904498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..............X...........................] duration: 36.932073938s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:47:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 36.932112871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX....] duration: 13.14927086s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:47:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 13.149300767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..............X..........................] duration: 7.479382433s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:46:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 7.479408301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..............X.........................] duration: 22.604321948s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:46:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 22.604341716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...] duration: 6.01106698s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:46:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.01109358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX..] duration: 10.858690919s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 10.858762625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..............X........................] duration: 13.843051043s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 13.843080349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.] duration: 27.383321548s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:45:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 27.383364249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX] duration: 30.002690585s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 30.002703109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..............X.......................] duration: 16.372425717s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 16.372453049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XX] duration: 30.002963659s trace: https://jaeger.dev.storj.io/trace/6a63853ece39ac5d FAIL (2025-12-26 20:44:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OJKLHZP4QUL6WC4CUBC4ALNNTS6BNLZAKDEIIUPZU4HNTUAMUYLQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 4TDWDS6V3R5QA2PZ6AZRTKWLVKRASW6Y5KU7KCKULSTUYJD5C26A) context deadline exceeded ======LOGS====== 26 20:43:31.827779558 (509h14m31.698862594s) [upload=446859]: Starting upload Dec 26 20:43:31.827834593 (509h14m31.698917628s) [upload=446859]: Done starting upload Dec 26 20:43:31.828014786 (509h14m31.699097821s) [upload=446859]: Uploading segments... Dec 26 20:43:31.828088216 (509h14m31.699171261s) [upload=446859]: Got next segment. Inline: false Dec 26 20:43:31.828110468 (509h14m31.699193503s) [upload=446859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:43:31.828114536 (509h14m31.699197581s) [upload=446859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:43:32.073213258 (509h14m31.944296293s) [seg_pos={0 0}, upload=446859]: Begin upload segment... Dec 26 20:43:32.073256430 (509h14m31.944339475s) [seg_pos={0 0}, upload=446859]: Done begin upload segment. Dec 26 20:43:32.073284083 (509h14m31.944367118s) [noise=true , address=145.79.227.46:4602 , piece_id=SJ6QBMVV26ICJIMHIZLCSIAXLRD45XSMOL3JAPCA4XWYMG4E3S7A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.073298760 (509h14m31.944381795s) [noise=true , address=45.91.116.12:4228 , piece_id=4TDWDS6V3R5QA2PZ6AZRTKWLVKRASW6Y5KU7KCKULSTUYJD5C26A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.073853016 (509h14m31.944936061s) [noise=true , address=145.79.227.36:4518 , piece_id=BMTNDHPAGDUHOMT5II55URT6YNLGHDIJTWDJZNLQ6OMBOD6VXM2A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.074085579 (509h14m31.945168614s) [noise=true , address=145.79.227.15:4874 , piece_id=ZC3YBACI5X6LSMLVSYDHP2JLGKJQY25FLUDWNF4BNQEAZRVKHLGA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.074954754 (509h14m31.946037789s) [noise=true , address=145.79.227.25:4618 , piece_id=25GETOMMHIJVJNDMMMXDGICN3KAD5UEGE2HQZS5V423ZLOXF735A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.075343073 (509h14m31.946426119s) [noise=true , address=145.79.227.17:4972 , piece_id=H3JJJ2IZHMLPCUKOBJWCWG4AF7FNWV7AJ3ZUKTDL5SGLDHYAMD3A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.075608128 (509h14m31.946691163s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZDVAHMG4KAP5TJZGC35WM2RXBADDZMJNBQ7AK7BH3LCKMWZ62FA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.075904713 (509h14m31.946987758s) [noise=true , address=45.91.116.11:4118 , piece_id=SY7SPKVSDSMHJVR2QXA3OGLN2QAVQS2MNDKJIQRLBO5I75M3WQ3A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076406608 (509h14m31.947489653s) [noise=true , address=77.90.31.16:4735 , piece_id=BYV2EM7YXVYAU5QNUFD7MY6MFJV6UVVWG5S7YEZBOWHDOJR7KPPQ, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076415315 (509h14m31.94749835s) [noise=true , address=45.91.116.6:4976 , piece_id=KK3H5QXVJM32VWJZ2UFGKJ6Q54OPCWGHY7JTCUHBD2EUKJLXVF5Q, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076456674 (509h14m31.947539709s) [noise=true , address=45.91.116.10:4083 , piece_id=UQZ3HBON554YMGZQ7QCWQYGHFOCHSCBFPHGVM7GAMCWPOS23Z3RA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076750192 (509h14m31.947833227s) [noise=true , address=45.91.116.4:4939 , piece_id=PGIEUHTDQUW7UPFLQRNKYJS6VKOYU2LLLGLSGXFRYP36YVS47HHA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076853990 (509h14m31.947937035s) [noise=true , address=77.90.31.10:4520 , piece_id=PE54A53EAKOZXPXVIO64IWKOMYIB5QGRSMCS4NGEGPBF74UOGKZQ, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.077033963 (509h14m31.948117008s) [noise=true , address=145.79.227.22:4509 , piece_id=JWZNBWRS23HVDOLTA3XCJYTUU6UMRCF6TCI43J7HISLLYD4YXI2A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.077428595 (509h14m31.94851164s) [noise=true , address=45.91.116.5:4297 , piece_id=OJKLHZP4QUL6WC4CUBC4ALNNTS6BNLZAKDEIIUPZU4HNTUAMUYLQ, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.077697115 (509h14m31.94878015s) [noise=true , address=145.79.227.52:4447 , piece_id=NIUE3X627GJQEW4II3TGQVOVSFCQDT4Y7VEKRC3IHTINO6FFRQVA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:33.786420078 (509h14m33.657503123s) [upload=446859]: Next returned nil segment Dec 26 20:43:33.786458861 (509h14m33.657541897s) [upload=446859]: Waiting for error group managing segments... Dec 26 20:43:34.071567109 (509h14m33.942650144s) [noise=true , address=145.79.227.15:4874 , piece_id=ZC3YBACI5X6LSMLVSYDHP2JLGKJQY25FLUDWNF4BNQEAZRVKHLGA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.076340007 (509h14m33.947423042s) [noise=true , address=145.79.227.46:4602 , piece_id=SJ6QBMVV26ICJIMHIZLCSIAXLRD45XSMOL3JAPCA4XWYMG4E3S7A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.078259822 (509h14m33.949342857s) [noise=true , address=145.79.227.25:4618 , piece_id=25GETOMMHIJVJNDMMMXDGICN3KAD5UEGE2HQZS5V423ZLOXF735A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.080296029 (509h14m33.951379074s) [noise=true , address=145.79.227.36:4518 , piece_id=BMTNDHPAGDUHOMT5II55URT6YNLGHDIJTWDJZNLQ6OMBOD6VXM2A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.098910434 (509h14m33.96999347s) [noise=true , address=145.79.227.52:4447 , piece_id=NIUE3X627GJQEW4II3TGQVOVSFCQDT4Y7VEKRC3IHTINO6FFRQVA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.108246703 (509h14m33.979329738s) [noise=true , address=145.79.227.22:4509 , piece_id=JWZNBWRS23HVDOLTA3XCJYTUU6UMRCF6TCI43J7HISLLYD4YXI2A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.322044521 (509h14m34.193127556s) [noise=true , address=45.91.116.5:4297 , piece_id=OJKLHZP4QUL6WC4CUBC4ALNNTS6BNLZAKDEIIUPZU4HNTUAMUYLQ, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.353763721 (509h14m34.224846756s) [noise=true , address=45.91.116.10:4083 , piece_id=UQZ3HBON554YMGZQ7QCWQYGHFOCHSCBFPHGVM7GAMCWPOS23Z3RA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.356366086 (509h14m34.227449121s) [noise=true , address=45.91.116.12:4228 , piece_id=4TDWDS6V3R5QA2PZ6AZRTKWLVKRASW6Y5KU7KCKULSTUYJD5C26A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.385228866 (509h14m34.256311911s) [noise=true , address=45.91.116.4:4939 , piece_id=PGIEUHTDQUW7UPFLQRNKYJS6VKOYU2LLLGLSGXFRYP36YVS47HHA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.459915914 (509h14m34.330998959s) [noise=true , address=145.79.227.17:4972 , piece_id=H3JJJ2IZHMLPCUKOBJWCWG4AF7FNWV7AJ3ZUKTDL5SGLDHYAMD3A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.932132016 (509h14m34.803215062s) [noise=true , address=77.90.31.10:4520 , piece_id=PE54A53EAKOZXPXVIO64IWKOMYIB5QGRSMCS4NGEGPBF74UOGKZQ, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:35.060598159 (509h14m34.931681194s) [noise=true , address=45.91.116.11:4118 , piece_id=SY7SPKVSDSMHJVR2QXA3OGLN2QAVQS2MNDKJIQRLBO5I75M3WQ3A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:35.060617015 (509h14m34.93170005s) [seg_pos={0 0}, upload=446859]: Segment reached optimal threshold of 13 pieces. Dec 26 20:43:35.060803269 (509h14m34.931886314s) [noise=true , address=77.90.31.16:4735 , piece_id=BYV2EM7YXVYAU5QNUFD7MY6MFJV6UVVWG5S7YEZBOWHDOJR7KPPQ, seg_pos={0 0}, upload=446859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 20:43:35.060809302 (509h14m34.931892337s) [noise=true , address=77.90.31.16:4735 , piece_id=BYV2EM7YXVYAU5QNUFD7MY6MFJV6UVVWG5S7YEZBOWHDOJR7KPPQ, seg_pos={0 0}, upload=446859]: Long tail cancellation due to optimal threshold Dec 26 20:43:35.060879435 (509h14m34.93196248s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZDVAHMG4KAP5TJZGC35WM2RXBADDZMJNBQ7AK7BH3LCKMWZ62FA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:33870->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:33870->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33870->145.79.227.5:4222: use of closed network connection Dec 26 20:43:35.060892069 (509h14m34.931975104s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZDVAHMG4KAP5TJZGC35WM2RXBADDZMJNBQ7AK7BH3LCKMWZ62FA, seg_pos={0 0}, upload=446859]: Long tail cancellation due to optimal threshold Dec 26 20:43:35.061057524 (509h14m34.932140559s) [noise=true , address=45.91.116.6:4976 , piece_id=KK3H5QXVJM32VWJZ2UFGKJ6Q54OPCWGHY7JTCUHBD2EUKJLXVF5Q, seg_pos={0 0}, upload=446859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 20:43:35.061065149 (509h14m34.932148184s) [noise=true , address=45.91.116.6:4976 , piece_id=KK3H5QXVJM32VWJZ2UFGKJ6Q54OPCWGHY7JTCUHBD2EUKJLXVF5Q, seg_pos={0 0}, upload=446859]: Long tail cancellation due to optimal threshold Dec 26 20:43:35.061083473 (509h14m34.932166518s) [seg_pos={0 0}, upload=446859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:43:35.061146283 (509h14m34.932229318s) [upload=446859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:43:35.061150911 (509h14m34.932233946s) [upload=446859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:43:35.358156253 (509h14m35.229239308s) [upload=446859]: Done uploading segments... Dec 26 20:43:35.358177122 (509h14m35.229260167s) [upload=446859]: Upload finished. err: <nil> Duration: 30.002991452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..............X......................] duration: 22.870797964s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 22.870826841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..X] duration: 30.002601407s trace: https://jaeger.dev.storj.io/trace/76a3eb206709086d FAIL (2025-12-26 20:43:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 20:42:53.452673042 (509h13m53.323756077s) [upload=446850]: Starting upload Dec 26 20:42:53.452749768 (509h13m53.323832803s) [upload=446850]: Done starting upload Dec 26 20:42:53.452808440 (509h13m53.323891475s) [upload=446850]: Uploading segments... Dec 26 20:42:53.452948256 (509h13m53.324031301s) [upload=446850]: Got next segment. Inline: false Dec 26 20:42:53.452985026 (509h13m53.324068061s) [upload=446850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:42:53.452989364 (509h13m53.324072399s) [upload=446850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:42:53.699109089 (509h13m53.570192124s) [seg_pos={0 0}, upload=446850]: Begin upload segment... Dec 26 20:42:53.699154026 (509h13m53.570237061s) [seg_pos={0 0}, upload=446850]: Done begin upload segment. Dec 26 20:42:53.699180455 (509h13m53.57026349s) [noise=true , address=45.91.116.6:4974 , piece_id=P4MI4VNKCMSC56YNO52E623IXIUAKP2LNBEG7T3OP5YNKKMX7BCQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.699750982 (509h13m53.570834017s) [noise=true , address=145.79.227.15:4874 , piece_id=O7UKVEQ5L6Q6X4EB3KBC2WPF6US4IF5ALLAAKRTGOGO3XHQ7YD3A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.700661506 (509h13m53.571744541s) [noise=true , address=145.79.227.48:4301 , piece_id=JX3SVJSZGJOI2RMLMUT7EVJNSXQ53BB4G5QMHEUREKNW23L4V3LQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.701146840 (509h13m53.572229885s) [noise=true , address=145.79.227.42:4123 , piece_id=SI2NCHXTYUN7MFEIWUDSPIUODD3O7JLCW6IZ5CPE5HVIC2EIX3LA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.701651050 (509h13m53.572734085s) [noise=true , address=145.79.227.47:4266 , piece_id=Q4KPGM7R3MBONTHUAG5S6CGR4BDWKDEJ2ZGST6H6IEQLFJSISV2A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.702201147 (509h13m53.573284182s) [noise=true , address=145.79.227.13:4804 , piece_id=WU7S7LN3IPTR2HFTSKLEXDMMQMYC4BDSCQKFIHKTXUH4YK3CBQTQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.703583289 (509h13m53.574666335s) [noise=true , address=77.90.31.10:4520 , piece_id=EVOR62CPL63GGJJQA6GXOA2WAW3PG32YXCRB77MFQNEVSDH4P24A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.704767635 (509h13m53.57585067s) [noise=true , address=145.79.227.36:4517 , piece_id=2HDK2GDQ3V7JG3SCR235TJE6AETIWD4TCXOIWH5K23WJGTB7WABA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.705300158 (509h13m53.576383193s) [noise=true , address=145.79.227.25:4618 , piece_id=S7VAIMY5CMJIU7PYO66XVBQGZIAYNWWIVQYEO5PCK4774A5CYBCA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.708118705 (509h13m53.579201761s) [noise=true , address=145.79.227.38:4978 , piece_id=7IUY2X6KBIRN4LXZ7AHJKDBCLFBSO2WGDHKE7DO2NLISAY5IJGGQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.709352265 (509h13m53.58043531s) [noise=true , address=45.91.116.10:4083 , piece_id=VZ7HEFNEDRYHBCLYVDGROGXN7KP4274HCA5KKMKC66LFDPWMQX4A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.712141386 (509h13m53.583224421s) [noise=true , address=145.79.227.44:4195 , piece_id=5M56H5P3H4W2PV3VFFPV33GT7TVSKHICJQ2BC5JOADZRKEIEDZDQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.713687509 (509h13m53.584770554s) [noise=true , address=45.91.116.12:4226 , piece_id=5RRTTAKVHXJNIW2B3EJNG654W76OKRP5SJT66UPTMYQ3RYBHNLFQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.715132111 (509h13m53.586215146s) [noise=true , address=145.79.227.4:4186 , piece_id=TVEWTXJF5P5HH53BHBNVUIU7EZ7LFG4XQYJEUZ46IHGOIWRBGBVA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.716635082 (509h13m53.587718117s) [noise=true , address=145.79.227.3:4150 , piece_id=KIBZZD5MLGRMOEZMFBQODTEDF7CPDTFRN74BQ3R7WVGAOA2ZY55Q, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.723786299 (509h13m53.594869344s) [noise=true , address=77.90.31.14:4661 , piece_id=PUXUG2OO77TB2VWYGABS4K7NMSFGLUG77VOBZKJDGNRGSFIDT7FQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:55.391698591 (509h13m55.262781626s) [upload=446850]: Next returned nil segment Dec 26 20:42:55.391712427 (509h13m55.262795462s) [upload=446850]: Waiting for error group managing segments... Dec 26 20:42:55.703427408 (509h13m55.574510453s) [noise=true , address=145.79.227.15:4874 , piece_id=O7UKVEQ5L6Q6X4EB3KBC2WPF6US4IF5ALLAAKRTGOGO3XHQ7YD3A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.713861356 (509h13m55.584944391s) [noise=true , address=145.79.227.47:4266 , piece_id=Q4KPGM7R3MBONTHUAG5S6CGR4BDWKDEJ2ZGST6H6IEQLFJSISV2A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.723816092 (509h13m55.594899127s) [noise=true , address=145.79.227.38:4978 , piece_id=7IUY2X6KBIRN4LXZ7AHJKDBCLFBSO2WGDHKE7DO2NLISAY5IJGGQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.724736855 (509h13m55.5958199s) [noise=true , address=145.79.227.48:4301 , piece_id=JX3SVJSZGJOI2RMLMUT7EVJNSXQ53BB4G5QMHEUREKNW23L4V3LQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.738045878 (509h13m55.609128913s) [noise=true , address=145.79.227.25:4618 , piece_id=S7VAIMY5CMJIU7PYO66XVBQGZIAYNWWIVQYEO5PCK4774A5CYBCA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.738454646 (509h13m55.609537691s) [noise=true , address=145.79.227.36:4517 , piece_id=2HDK2GDQ3V7JG3SCR235TJE6AETIWD4TCXOIWH5K23WJGTB7WABA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.739559450 (509h13m55.610642485s) [noise=true , address=145.79.227.42:4123 , piece_id=SI2NCHXTYUN7MFEIWUDSPIUODD3O7JLCW6IZ5CPE5HVIC2EIX3LA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.767277460 (509h13m55.638360495s) [noise=true , address=145.79.227.13:4804 , piece_id=WU7S7LN3IPTR2HFTSKLEXDMMQMYC4BDSCQKFIHKTXUH4YK3CBQTQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.986559662 (509h13m55.857642697s) [noise=true , address=45.91.116.10:4083 , piece_id=VZ7HEFNEDRYHBCLYVDGROGXN7KP4274HCA5KKMKC66LFDPWMQX4A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:56.009593071 (509h13m55.880676116s) [noise=true , address=45.91.116.12:4226 , piece_id=5RRTTAKVHXJNIW2B3EJNG654W76OKRP5SJT66UPTMYQ3RYBHNLFQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:56.014253576 (509h13m55.885336611s) [noise=true , address=77.90.31.10:4520 , piece_id=EVOR62CPL63GGJJQA6GXOA2WAW3PG32YXCRB77MFQNEVSDH4P24A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:56.194939815 (509h13m56.06602286s) [noise=true , address=77.90.31.14:4661 , piece_id=PUXUG2OO77TB2VWYGABS4K7NMSFGLUG77VOBZKJDGNRGSFIDT7FQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:57.019585927 (509h13m56.890668972s) [noise=true , address=45.91.116.6:4974 , piece_id=P4MI4VNKCMSC56YNO52E623IXIUAKP2LNBEG7T3OP5YNKKMX7BCQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:57.019604753 (509h13m56.890687798s) [seg_pos={0 0}, upload=446850]: Segment reached optimal threshold of 13 pieces. Dec 26 20:42:57.019808722 (509h13m56.890891787s) [noise=true , address=145.79.227.3:4150 , piece_id=KIBZZD5MLGRMOEZMFBQODTEDF7CPDTFRN74BQ3R7WVGAOA2ZY55Q, seg_pos={0 0}, upload=446850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 26 20:42:57.019819462 (509h13m56.890902507s) [noise=true , address=145.79.227.3:4150 , piece_id=KIBZZD5MLGRMOEZMFBQODTEDF7CPDTFRN74BQ3R7WVGAOA2ZY55Q, seg_pos={0 0}, upload=446850]: Long tail cancellation due to optimal threshold Dec 26 20:42:57.020170470 (509h13m56.891253515s) [noise=true , address=145.79.227.44:4195 , piece_id=5M56H5P3H4W2PV3VFFPV33GT7TVSKHICJQ2BC5JOADZRKEIEDZDQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 20:42:57.020178455 (509h13m56.89126149s) [noise=true , address=145.79.227.44:4195 , piece_id=5M56H5P3H4W2PV3VFFPV33GT7TVSKHICJQ2BC5JOADZRKEIEDZDQ, seg_pos={0 0}, upload=446850]: Long tail cancellation due to optimal threshold Dec 26 20:42:57.020336396 (509h13m56.891419441s) [noise=true , address=145.79.227.4:4186 , piece_id=TVEWTXJF5P5HH53BHBNVUIU7EZ7LFG4XQYJEUZ46IHGOIWRBGBVA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:57302->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57302->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->145.79.227.4:4186: use of closed network connection Dec 26 20:42:57.020345123 (509h13m56.891428168s) [noise=true , address=145.79.227.4:4186 , piece_id=TVEWTXJF5P5HH53BHBNVUIU7EZ7LFG4XQYJEUZ46IHGOIWRBGBVA, seg_pos={0 0}, upload=446850]: Long tail cancellation due to optimal threshold Dec 26 20:42:57.020366534 (509h13m56.891449579s) [seg_pos={0 0}, upload=446850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:42:57.020442278 (509h13m56.891525323s) [upload=446850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:42:57.020447237 (509h13m56.891530272s) [upload=446850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:42:57.306538346 (509h13m57.177621401s) [upload=446850]: Done uploading segments... Dec 26 20:42:57.306563945 (509h13m57.17764699s) [upload=446850]: Upload finished. err: <nil> Duration: 30.002624461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..............X.....................] duration: 30.345046084s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:43:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 30.345063429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..............X....................] duration: 15.510529434s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:42:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 15.510655675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..] duration: 6.210911421s trace: https://jaeger.dev.storj.io/trace/61359e2ec8c725f4 FAIL (2025-12-26 20:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ) context deadline exceeded ======LOGS====== segments... Dec 26 20:41:18.018143223 (509h12m17.889226278s) [upload=446828]: Got next segment. Inline: false Dec 26 20:41:18.018170776 (509h12m17.889253811s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:41:18.018175695 (509h12m17.88925874s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:41:18.268182603 (509h12m18.139265638s) [seg_pos={0 0}, upload=446828]: Begin upload segment... Dec 26 20:41:18.268227127 (509h12m18.139310172s) [seg_pos={0 0}, upload=446828]: Done begin upload segment. Dec 26 20:41:18.268252105 (509h12m18.13933514s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268324282 (509h12m18.139407317s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268882405 (509h12m18.13996545s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269337542 (509h12m18.140420587s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269733095 (509h12m18.14081613s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270201988 (509h12m18.141285023s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270705347 (509h12m18.141788382s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.271129855 (509h12m18.1422129s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.272047201 (509h12m18.143130236s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273169739 (509h12m18.144252784s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273302792 (509h12m18.144385837s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276011821 (509h12m18.147094856s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276174972 (509h12m18.147258007s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.279685707 (509h12m18.150768752s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.283723977 (509h12m18.154807022s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.284762333 (509h12m18.155845378s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:19.988111673 (509h12m19.859194738s) [upload=446828]: Next returned nil segment Dec 26 20:41:19.988143443 (509h12m19.859226488s) [upload=446828]: Waiting for error group managing segments... Dec 26 20:41:20.276877755 (509h12m20.14796079s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.278237724 (509h12m20.149320759s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.279899659 (509h12m20.150982694s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.281332086 (509h12m20.152415131s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.290793022 (509h12m20.161876077s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.292610823 (509h12m20.163693868s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.296464041 (509h12m20.167547086s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.307126443 (509h12m20.178209478s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.404347300 (509h12m20.275430335s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.453137093 (509h12m20.324220128s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.501363175 (509h12m20.37244621s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.524892048 (509h12m20.395975093s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615609551 (509h12m20.486692586s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615633336 (509h12m20.486716371s) [seg_pos={0 0}, upload=446828]: Segment reached optimal threshold of 13 pieces. Dec 26 20:41:20.615844709 (509h12m20.486927744s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 20:41:20.615852433 (509h12m20.486935478s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.615990236 (509h12m20.487073281s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection Dec 26 20:41:20.615996357 (509h12m20.487079402s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616372924 (509h12m20.487455969s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection Dec 26 20:41:20.616393494 (509h12m20.487476539s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616450602 (509h12m20.487533647s) [seg_pos={0 0}, upload=446828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:41:20.616640264 (509h12m20.487723309s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:41:20.616649211 (509h12m20.487732246s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:41:20.920505607 (509h12m20.791588662s) [upload=446828]: Done uploading segments... Dec 26 20:41:20.920531566 (509h12m20.791614611s) [upload=446828]: Upload finished. err: <nil> Duration: 6.210932732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..............X...................] duration: 22.335169513s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 22.335195602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX.] duration: 5.985062502s trace: https://jaeger.dev.storj.io/trace/61359e2ec8c725f4 FAIL (2025-12-26 20:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ) context deadline exceeded ======LOGS====== segments... Dec 26 20:41:18.018143223 (509h12m17.889226278s) [upload=446828]: Got next segment. Inline: false Dec 26 20:41:18.018170776 (509h12m17.889253811s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:41:18.018175695 (509h12m17.88925874s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:41:18.268182603 (509h12m18.139265638s) [seg_pos={0 0}, upload=446828]: Begin upload segment... Dec 26 20:41:18.268227127 (509h12m18.139310172s) [seg_pos={0 0}, upload=446828]: Done begin upload segment. Dec 26 20:41:18.268252105 (509h12m18.13933514s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268324282 (509h12m18.139407317s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268882405 (509h12m18.13996545s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269337542 (509h12m18.140420587s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269733095 (509h12m18.14081613s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270201988 (509h12m18.141285023s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270705347 (509h12m18.141788382s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.271129855 (509h12m18.1422129s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.272047201 (509h12m18.143130236s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273169739 (509h12m18.144252784s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273302792 (509h12m18.144385837s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276011821 (509h12m18.147094856s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276174972 (509h12m18.147258007s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.279685707 (509h12m18.150768752s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.283723977 (509h12m18.154807022s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.284762333 (509h12m18.155845378s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:19.988111673 (509h12m19.859194738s) [upload=446828]: Next returned nil segment Dec 26 20:41:19.988143443 (509h12m19.859226488s) [upload=446828]: Waiting for error group managing segments... Dec 26 20:41:20.276877755 (509h12m20.14796079s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.278237724 (509h12m20.149320759s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.279899659 (509h12m20.150982694s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.281332086 (509h12m20.152415131s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.290793022 (509h12m20.161876077s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.292610823 (509h12m20.163693868s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.296464041 (509h12m20.167547086s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.307126443 (509h12m20.178209478s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.404347300 (509h12m20.275430335s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.453137093 (509h12m20.324220128s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.501363175 (509h12m20.37244621s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.524892048 (509h12m20.395975093s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615609551 (509h12m20.486692586s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615633336 (509h12m20.486716371s) [seg_pos={0 0}, upload=446828]: Segment reached optimal threshold of 13 pieces. Dec 26 20:41:20.615844709 (509h12m20.486927744s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 20:41:20.615852433 (509h12m20.486935478s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.615990236 (509h12m20.487073281s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection Dec 26 20:41:20.615996357 (509h12m20.487079402s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616372924 (509h12m20.487455969s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection Dec 26 20:41:20.616393494 (509h12m20.487476539s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616450602 (509h12m20.487533647s) [seg_pos={0 0}, upload=446828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:41:20.616640264 (509h12m20.487723309s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:41:20.616649211 (509h12m20.487732246s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:41:20.920505607 (509h12m20.791588662s) [upload=446828]: Done uploading segments... Dec 26 20:41:20.920531566 (509h12m20.791614611s) [upload=446828]: Upload finished. err: <nil> Duration: 5.985081719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX] duration: 30.003450872s trace: https://jaeger.dev.storj.io/trace/61359e2ec8c725f4 FAIL (2025-12-26 20:41:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ) context deadline exceeded ======LOGS====== segments... Dec 26 20:41:18.018143223 (509h12m17.889226278s) [upload=446828]: Got next segment. Inline: false Dec 26 20:41:18.018170776 (509h12m17.889253811s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:41:18.018175695 (509h12m17.88925874s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:41:18.268182603 (509h12m18.139265638s) [seg_pos={0 0}, upload=446828]: Begin upload segment... Dec 26 20:41:18.268227127 (509h12m18.139310172s) [seg_pos={0 0}, upload=446828]: Done begin upload segment. Dec 26 20:41:18.268252105 (509h12m18.13933514s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268324282 (509h12m18.139407317s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268882405 (509h12m18.13996545s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269337542 (509h12m18.140420587s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269733095 (509h12m18.14081613s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270201988 (509h12m18.141285023s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270705347 (509h12m18.141788382s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.271129855 (509h12m18.1422129s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.272047201 (509h12m18.143130236s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273169739 (509h12m18.144252784s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273302792 (509h12m18.144385837s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276011821 (509h12m18.147094856s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276174972 (509h12m18.147258007s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.279685707 (509h12m18.150768752s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.283723977 (509h12m18.154807022s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.284762333 (509h12m18.155845378s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:19.988111673 (509h12m19.859194738s) [upload=446828]: Next returned nil segment Dec 26 20:41:19.988143443 (509h12m19.859226488s) [upload=446828]: Waiting for error group managing segments... Dec 26 20:41:20.276877755 (509h12m20.14796079s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.278237724 (509h12m20.149320759s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.279899659 (509h12m20.150982694s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.281332086 (509h12m20.152415131s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.290793022 (509h12m20.161876077s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.292610823 (509h12m20.163693868s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.296464041 (509h12m20.167547086s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.307126443 (509h12m20.178209478s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.404347300 (509h12m20.275430335s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.453137093 (509h12m20.324220128s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.501363175 (509h12m20.37244621s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.524892048 (509h12m20.395975093s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615609551 (509h12m20.486692586s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615633336 (509h12m20.486716371s) [seg_pos={0 0}, upload=446828]: Segment reached optimal threshold of 13 pieces. Dec 26 20:41:20.615844709 (509h12m20.486927744s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 20:41:20.615852433 (509h12m20.486935478s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.615990236 (509h12m20.487073281s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection Dec 26 20:41:20.615996357 (509h12m20.487079402s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616372924 (509h12m20.487455969s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection Dec 26 20:41:20.616393494 (509h12m20.487476539s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616450602 (509h12m20.487533647s) [seg_pos={0 0}, upload=446828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:41:20.616640264 (509h12m20.487723309s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:41:20.616649211 (509h12m20.487732246s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:41:20.920505607 (509h12m20.791588662s) [upload=446828]: Done uploading segments... Dec 26 20:41:20.920531566 (509h12m20.791614611s) [upload=446828]: Upload finished. err: <nil> Duration: 30.00348068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..............X..................] duration: 31.159927842s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:41:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 31.159951747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.X] duration: 30.002294024s trace: https://jaeger.dev.storj.io/trace/27ac1dc1ce614769 FAIL (2025-12-26 20:41:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1522 (509h11m42.833344557s) [upload=446820]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:40:43.211387092 (509h11m43.082470137s) [seg_pos={0 0}, upload=446820]: Begin upload segment... Dec 26 20:40:43.211443689 (509h11m43.082526734s) [seg_pos={0 0}, upload=446820]: Done begin upload segment. Dec 26 20:40:43.211476922 (509h11m43.082559967s) [noise=true , address=145.79.227.48:4302 , piece_id=REW45FGQLXKW6WWY4ZCSBWR2GMFHR7MSIMHZQ2FRDOFEZBDVPE5A, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.211499305 (509h11m43.08258235s) [noise=true , address=77.90.31.10:4520 , piece_id=O4ITDULB7XDQ4NKZVN4YZS5GFBEIQWCC2CBRHFFT64GZZFOJ2URA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.212051827 (509h11m43.083134862s) [noise=true , address=145.79.227.5:4222 , piece_id=Q4PK7NXTVZYJ6XB64SOREQUXBGOU5GH6FVJOJQH7XPBTDC5JWKJQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.212418826 (509h11m43.083501861s) [noise=true , address=45.91.116.10:4082 , piece_id=K3NBD25BBARUWDZNIZ4XNICBN7QYNNIWN4Y6C2US4LPDBMZOLXRQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.212783811 (509h11m43.083866856s) [noise=true , address=145.79.227.36:4517 , piece_id=BHUS4UP2EVEVAN7WDCFGG7M4VQHFRDUIWPSB4SRA3PQ427LAKZTQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.213151511 (509h11m43.084234556s) [noise=true , address=45.91.116.6:4975 , piece_id=4HIA72JC65B3XXCN3JLLQDN3SWHVQQLXMAST5CWFTVL4KCW7QM5A, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.213555791 (509h11m43.084638836s) [noise=true , address=45.91.116.13:4153 , piece_id=WI7ZN2RW3BLARRAK5HPLMFYG4KSM33TSLTX5YANIHQFWWHPZVUSA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.214484630 (509h11m43.085567665s) [noise=true , address=145.79.227.41:4086 , piece_id=ZYEKA24NWBT5LIL44KEDW2AF6UDDLHJPJNGKY2A5GKRZLKRQVFIQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.214784019 (509h11m43.085867064s) [noise=true , address=77.90.31.17:4773 , piece_id=Q45DXROJS756ZMPYB4DSGGC2B7VFHM3Y477MEI3DKOJDW2LYTYWQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.215371789 (509h11m43.086454824s) [noise=true , address=145.79.227.45:4230 , piece_id=7NOZJPCAHQXGMBNZVGDPC6FD4YJGUVB6XYHMIAXJZEQSMIH6LJFA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.215776399 (509h11m43.086859435s) [noise=true , address=145.79.227.11:4730 , piece_id=T3QT2JV6LOQSWS5WVLGBYGX4XFKVPC5H3EXFRUYM4LPSQTF3HTVQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.216616659 (509h11m43.087699704s) [noise=true , address=145.79.227.46:4601 , piece_id=3X5YTJUYGKNHLR7BAU7XKIPZJ6AL6ZU7CCHD6YJ6S54IC45HNMLQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.217114858 (509h11m43.088197893s) [noise=true , address=145.79.227.49:4339 , piece_id=4LXJMUSX75BU35N2HDZGUWQ4A44XMO2PKFHRKUKLMCMPXRE6X7GA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.218015643 (509h11m43.089098678s) [noise=true , address=145.79.227.6:4259 , piece_id=6QTBYCHQIKYTRRW7KBQ75Q7E4D4L23CAIZJEDZTXPJZLXRQCJDIQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.219325618 (509h11m43.090408653s) [noise=true , address=145.79.227.52:4446 , piece_id=EUGTKXFSSZAIVO2LDNM6NOSI2CXJTW2RYO4A6ZZTFELPBNH2AM2A, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.220749509 (509h11m43.091832554s) [noise=true , address=45.91.116.8:4047 , piece_id=733PERYUX63YGKH7LIJE2BJQUAS5XE4IQ7IBO36DBHTIBXOE2MFQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:44.930623463 (509h11m44.801706498s) [upload=446820]: Next returned nil segment Dec 26 20:40:44.930638151 (509h11m44.801721186s) [upload=446820]: Waiting for error group managing segments... Dec 26 20:40:45.210619666 (509h11m45.081702711s) [noise=true , address=145.79.227.49:4339 , piece_id=4LXJMUSX75BU35N2HDZGUWQ4A44XMO2PKFHRKUKLMCMPXRE6X7GA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.217776223 (509h11m45.088859258s) [noise=true , address=145.79.227.48:4302 , piece_id=REW45FGQLXKW6WWY4ZCSBWR2GMFHR7MSIMHZQ2FRDOFEZBDVPE5A, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.224708554 (509h11m45.095791589s) [noise=true , address=145.79.227.45:4230 , piece_id=7NOZJPCAHQXGMBNZVGDPC6FD4YJGUVB6XYHMIAXJZEQSMIH6LJFA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.228149926 (509h11m45.099232961s) [noise=true , address=145.79.227.36:4517 , piece_id=BHUS4UP2EVEVAN7WDCFGG7M4VQHFRDUIWPSB4SRA3PQ427LAKZTQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.228965359 (509h11m45.100048394s) [noise=true , address=145.79.227.46:4601 , piece_id=3X5YTJUYGKNHLR7BAU7XKIPZJ6AL6ZU7CCHD6YJ6S54IC45HNMLQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.230541060 (509h11m45.101624095s) [noise=true , address=145.79.227.11:4730 , piece_id=T3QT2JV6LOQSWS5WVLGBYGX4XFKVPC5H3EXFRUYM4LPSQTF3HTVQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.256049983 (509h11m45.127133018s) [noise=true , address=145.79.227.52:4446 , piece_id=EUGTKXFSSZAIVO2LDNM6NOSI2CXJTW2RYO4A6ZZTFELPBNH2AM2A, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.319631223 (509h11m45.190714268s) [noise=true , address=145.79.227.6:4259 , piece_id=6QTBYCHQIKYTRRW7KBQ75Q7E4D4L23CAIZJEDZTXPJZLXRQCJDIQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.460268812 (509h11m45.331351858s) [noise=true , address=45.91.116.10:4082 , piece_id=K3NBD25BBARUWDZNIZ4XNICBN7QYNNIWN4Y6C2US4LPDBMZOLXRQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.513139858 (509h11m45.384222903s) [noise=true , address=77.90.31.10:4520 , piece_id=O4ITDULB7XDQ4NKZVN4YZS5GFBEIQWCC2CBRHFFT64GZZFOJ2URA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.522104709 (509h11m45.393187754s) [noise=true , address=45.91.116.13:4153 , piece_id=WI7ZN2RW3BLARRAK5HPLMFYG4KSM33TSLTX5YANIHQFWWHPZVUSA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.528016115 (509h11m45.39909916s) [noise=true , address=45.91.116.8:4047 , piece_id=733PERYUX63YGKH7LIJE2BJQUAS5XE4IQ7IBO36DBHTIBXOE2MFQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.568645140 (509h11m45.439728175s) [noise=true , address=45.91.116.6:4975 , piece_id=4HIA72JC65B3XXCN3JLLQDN3SWHVQQLXMAST5CWFTVL4KCW7QM5A, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.568665869 (509h11m45.439748904s) [seg_pos={0 0}, upload=446820]: Segment reached optimal threshold of 13 pieces. Dec 26 20:40:45.568876941 (509h11m45.439959976s) [noise=true , address=145.79.227.41:4086 , piece_id=ZYEKA24NWBT5LIL44KEDW2AF6UDDLHJPJNGKY2A5GKRZLKRQVFIQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33010->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33010->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33010->145.79.227.41:4086: use of closed network connection Dec 26 20:40:45.568883745 (509h11m45.43996678s) [noise=true , address=145.79.227.41:4086 , piece_id=ZYEKA24NWBT5LIL44KEDW2AF6UDDLHJPJNGKY2A5GKRZLKRQVFIQ, seg_pos={0 0}, upload=446820]: Long tail cancellation due to optimal threshold Dec 26 20:40:45.568894275 (509h11m45.43997731s) [noise=true , address=145.79.227.5:4222 , piece_id=Q4PK7NXTVZYJ6XB64SOREQUXBGOU5GH6FVJOJQH7XPBTDC5JWKJQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:43704->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:43704->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43704->145.79.227.5:4222: use of closed network connection Dec 26 20:40:45.568902871 (509h11m45.439985906s) [noise=true , address=145.79.227.5:4222 , piece_id=Q4PK7NXTVZYJ6XB64SOREQUXBGOU5GH6FVJOJQH7XPBTDC5JWKJQ, seg_pos={0 0}, upload=446820]: Long tail cancellation due to optimal threshold Dec 26 20:40:45.569000908 (509h11m45.440083943s) [noise=true , address=77.90.31.17:4773 , piece_id=Q45DXROJS756ZMPYB4DSGGC2B7VFHM3Y477MEI3DKOJDW2LYTYWQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:57046->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:57046->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57046->77.90.31.17:4773: use of closed network connection Dec 26 20:40:45.569007911 (509h11m45.440090946s) [noise=true , address=77.90.31.17:4773 , piece_id=Q45DXROJS756ZMPYB4DSGGC2B7VFHM3Y477MEI3DKOJDW2LYTYWQ, seg_pos={0 0}, upload=446820]: Long tail cancellation due to optimal threshold Dec 26 20:40:45.569035594 (509h11m45.440118629s) [seg_pos={0 0}, upload=446820]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:40:45.569117159 (509h11m45.440200194s) [upload=446820]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:40:45.569122008 (509h11m45.440205053s) [upload=446820]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:40:45.858244249 (509h11m45.729327284s) [upload=446820]: Done uploading segments... Dec 26 20:40:45.858273784 (509h11m45.729356819s) [upload=446820]: Upload finished. err: <nil> Duration:* 30.002315424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..............X.................] duration: 8.888216285s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 8.888230843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..............X................] duration: 19.543475972s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:40:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 19.543500878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.] duration: 7.938063349s trace: https://jaeger.dev.storj.io/trace/22c49e43bc74ae6b FAIL (2025-12-26 20:39:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:38:35.571057322 (509h9m35.442140367s) [upload=446789]: Starting upload Dec 26 20:38:35.571111015 (509h9m35.44219405s) [upload=446789]: Done starting upload Dec 26 20:38:35.571187500 (509h9m35.442270545s) [upload=446789]: Uploading segments... Dec 26 20:38:35.571339760 (509h9m35.442422805s) [upload=446789]: Got next segment. Inline: false Dec 26 20:38:35.571387732 (509h9m35.442470767s) [upload=446789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:38:35.571392571 (509h9m35.442475606s) [upload=446789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:38:35.815715693 (509h9m35.686798728s) [seg_pos={0 0}, upload=446789]: Begin upload segment... Dec 26 20:38:35.815757282 (509h9m35.686840317s) [seg_pos={0 0}, upload=446789]: Done begin upload segment. Dec 26 20:38:35.815817386 (509h9m35.686900421s) [noise=true , address=145.79.227.17:4971 , piece_id=UVNF63SU6VQS2BM7QNJRO7HOW3RZMXOBPXNIZSUI32TRM564J6LQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.815838968 (509h9m35.686922003s) [noise=true , address=145.79.227.38:4978 , piece_id=HWU4OBQOUZBBZQ5PUBHTAJ6KYB4FAFL5LN3SU7RTO4HYUIXIJMSA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.815912187 (509h9m35.686995222s) [noise=true , address=77.90.31.13:4627 , piece_id=OHSL7RCQ4V65L3VM2ICIWUOUNZQYXBSEBEVB5VDI6KGG6WMZFCQQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.816323450 (509h9m35.687406485s) [noise=true , address=145.79.227.6:4259 , piece_id=JI7CPNGZPWJSQOZ4EN6TTHDAGVGUXTJTFKU3BVUR6ZFQUEMXEGGQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.816612370 (509h9m35.687695405s) [noise=true , address=145.79.227.48:4301 , piece_id=BF442XW5DGF53AIULYAYEURTZU5AJHOLBUVYXN3XYY6LWJT7TBOQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.816617861 (509h9m35.687700896s) [noise=true , address=145.79.227.2:4114 , piece_id=SL57BDKDL2IZ6F7YDJY26BZYEYO3MEP7VTPZDY62P7K3GNXZ73QA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817008805 (509h9m35.68809184s) [noise=true , address=145.79.227.50:4375 , piece_id=PZ3HUBDKUYPXBTDNEITVAFA7NKIHU7S5AYL4Y72TYXGCAAMBNEOQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817039053 (509h9m35.688122098s) [noise=true , address=145.79.227.21:4475 , piece_id=3WKC6HE5KF2BQEVOF5I53M4CH2HIY5S2RE3V7SSDJKTTKXYWTWNQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817307664 (509h9m35.688390699s) [noise=true , address=145.79.227.49:4339 , piece_id=Y4CWHTCNEVH7SSZIGNYNX6IBFHNGRY45QIUJJDTNAI5YVHMIZWBQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817451137 (509h9m35.688534173s) [noise=true , address=145.79.227.12:4767 , piece_id=5SPLGOZWUMC5XZWDOJP3SIHAP7JVURGEWN6X22TIKSPTRSC4E3BA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817575524 (509h9m35.688658559s) [noise=true , address=145.79.227.36:4518 , piece_id=QAX3TOLUZGHU3FUTQ7EPZZ3HQLD46EZKFXGLUKNOJ3OSVHL7OXSQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817901034 (509h9m35.688984069s) [noise=true , address=145.79.227.15:4874 , piece_id=EKZSIIAMPHW5EK2QZJWLBTILRAR3FLL2KJFHIOACDOYQIWTQKBDA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817946000 (509h9m35.689029035s) [noise=true , address=45.91.116.8:4047 , piece_id=F4RHDDCZKUXO3SDSC76PSMQCT2KIT5T2SBUQKSOM2R5AMWIQWJVA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.818155989 (509h9m35.689239024s) [noise=true , address=145.79.227.41:4087 , piece_id=X2LULE7OTX5S62ZZSRED3TVKKTCV4BPYF4XNGTMVS57YIQH3EUAQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.818235951 (509h9m35.689318986s) [noise=true , address=145.79.227.4:4185 , piece_id=7LMP5WY6ZLG2JO5GICZ25JRNMJAL4L2ZADWBKI57MTRVL3Q2TDEA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.818618720 (509h9m35.689701755s) [noise=true , address=45.91.116.10:4084 , piece_id=JY5EP3XRI7JVPU7AH4FY6GQCV52QQ73JHQ73MQVPCG7GHLGLXMWQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:37.434486949 (509h9m37.305569994s) [upload=446789]: Next returned nil segment Dec 26 20:38:37.434508218 (509h9m37.305591254s) [upload=446789]: Waiting for error group managing segments... Dec 26 20:38:37.817540224 (509h9m37.688623269s) [noise=true , address=145.79.227.48:4301 , piece_id=BF442XW5DGF53AIULYAYEURTZU5AJHOLBUVYXN3XYY6LWJT7TBOQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.823210662 (509h9m37.694293697s) [noise=true , address=145.79.227.49:4339 , piece_id=Y4CWHTCNEVH7SSZIGNYNX6IBFHNGRY45QIUJJDTNAI5YVHMIZWBQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.833403761 (509h9m37.704486796s) [noise=true , address=145.79.227.15:4874 , piece_id=EKZSIIAMPHW5EK2QZJWLBTILRAR3FLL2KJFHIOACDOYQIWTQKBDA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.833893904 (509h9m37.704976939s) [noise=true , address=145.79.227.50:4375 , piece_id=PZ3HUBDKUYPXBTDNEITVAFA7NKIHU7S5AYL4Y72TYXGCAAMBNEOQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.833929802 (509h9m37.705012837s) [noise=true , address=145.79.227.12:4767 , piece_id=5SPLGOZWUMC5XZWDOJP3SIHAP7JVURGEWN6X22TIKSPTRSC4E3BA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.835668072 (509h9m37.706751117s) [noise=true , address=145.79.227.36:4518 , piece_id=QAX3TOLUZGHU3FUTQ7EPZZ3HQLD46EZKFXGLUKNOJ3OSVHL7OXSQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.836626337 (509h9m37.707709382s) [noise=true , address=145.79.227.21:4475 , piece_id=3WKC6HE5KF2BQEVOF5I53M4CH2HIY5S2RE3V7SSDJKTTKXYWTWNQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.850227766 (509h9m37.721310801s) [noise=true , address=145.79.227.38:4978 , piece_id=HWU4OBQOUZBBZQ5PUBHTAJ6KYB4FAFL5LN3SU7RTO4HYUIXIJMSA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:38.112495016 (509h9m37.983578081s) [noise=true , address=45.91.116.8:4047 , piece_id=F4RHDDCZKUXO3SDSC76PSMQCT2KIT5T2SBUQKSOM2R5AMWIQWJVA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:38.144925769 (509h9m38.016008804s) [noise=true , address=77.90.31.13:4627 , piece_id=OHSL7RCQ4V65L3VM2ICIWUOUNZQYXBSEBEVB5VDI6KGG6WMZFCQQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:38.186619000 (509h9m38.057702035s) [noise=true , address=45.91.116.10:4084 , piece_id=JY5EP3XRI7JVPU7AH4FY6GQCV52QQ73JHQ73MQVPCG7GHLGLXMWQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:39.384183618 (509h9m39.255266663s) [noise=true , address=145.79.227.4:4185 , piece_id=7LMP5WY6ZLG2JO5GICZ25JRNMJAL4L2ZADWBKI57MTRVL3Q2TDEA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:39.653749516 (509h9m39.524832552s) [noise=true , address=145.79.227.17:4971 , piece_id=UVNF63SU6VQS2BM7QNJRO7HOW3RZMXOBPXNIZSUI32TRM564J6LQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:39.653770356 (509h9m39.524853401s) [seg_pos={0 0}, upload=446789]: Segment reached optimal threshold of 13 pieces. Dec 26 20:38:39.653972170 (509h9m39.525055205s) [noise=true , address=145.79.227.2:4114 , piece_id=SL57BDKDL2IZ6F7YDJY26BZYEYO3MEP7VTPZDY62P7K3GNXZ73QA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:46682->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:46682->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46682->145.79.227.2:4114: use of closed network connection Dec 26 20:38:39.653977811 (509h9m39.525060856s) [noise=true , address=145.79.227.2:4114 , piece_id=SL57BDKDL2IZ6F7YDJY26BZYEYO3MEP7VTPZDY62P7K3GNXZ73QA, seg_pos={0 0}, upload=446789]: Long tail cancellation due to optimal threshold Dec 26 20:38:39.654120944 (509h9m39.525203979s) [noise=true , address=145.79.227.41:4087 , piece_id=X2LULE7OTX5S62ZZSRED3TVKKTCV4BPYF4XNGTMVS57YIQH3EUAQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 20:38:39.654164707 (509h9m39.525247752s) [noise=true , address=145.79.227.41:4087 , piece_id=X2LULE7OTX5S62ZZSRED3TVKKTCV4BPYF4XNGTMVS57YIQH3EUAQ, seg_pos={0 0}, upload=446789]: Long tail cancellation due to optimal threshold Dec 26 20:38:39.654198221 (509h9m39.525281256s) [noise=true , address=145.79.227.6:4259 , piece_id=JI7CPNGZPWJSQOZ4EN6TTHDAGVGUXTJTFKU3BVUR6ZFQUEMXEGGQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Dec 26 20:38:39.654205044 (509h9m39.525288089s) [noise=true , address=145.79.227.6:4259 , piece_id=JI7CPNGZPWJSQOZ4EN6TTHDAGVGUXTJTFKU3BVUR6ZFQUEMXEGGQ, seg_pos={0 0}, upload=446789]: Long tail cancellation due to optimal threshold Dec 26 20:38:39.654368305 (509h9m39.52545135s) [seg_pos={0 0}, upload=446789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:38:39.654477302 (509h9m39.525560347s) [upload=446789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:38:39.654483324 (509h9m39.525566369s) [upload=446789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:38:39.941361207 (509h9m39.812444242s) [upload=446789]: Done uploading segments... Dec 26 20:38:39.941379090 (509h9m39.812462125s) [upload=446789]: Upload finished. err: <nil> Duration: 7.938113334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X] duration: 30.002545116s trace: https://jaeger.dev.storj.io/trace/22c49e43bc74ae6b FAIL (2025-12-26 20:39:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:38:35.571057322 (509h9m35.442140367s) [upload=446789]: Starting upload Dec 26 20:38:35.571111015 (509h9m35.44219405s) [upload=446789]: Done starting upload Dec 26 20:38:35.571187500 (509h9m35.442270545s) [upload=446789]: Uploading segments... Dec 26 20:38:35.571339760 (509h9m35.442422805s) [upload=446789]: Got next segment. Inline: false Dec 26 20:38:35.571387732 (509h9m35.442470767s) [upload=446789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:38:35.571392571 (509h9m35.442475606s) [upload=446789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:38:35.815715693 (509h9m35.686798728s) [seg_pos={0 0}, upload=446789]: Begin upload segment... Dec 26 20:38:35.815757282 (509h9m35.686840317s) [seg_pos={0 0}, upload=446789]: Done begin upload segment. Dec 26 20:38:35.815817386 (509h9m35.686900421s) [noise=true , address=145.79.227.17:4971 , piece_id=UVNF63SU6VQS2BM7QNJRO7HOW3RZMXOBPXNIZSUI32TRM564J6LQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.815838968 (509h9m35.686922003s) [noise=true , address=145.79.227.38:4978 , piece_id=HWU4OBQOUZBBZQ5PUBHTAJ6KYB4FAFL5LN3SU7RTO4HYUIXIJMSA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.815912187 (509h9m35.686995222s) [noise=true , address=77.90.31.13:4627 , piece_id=OHSL7RCQ4V65L3VM2ICIWUOUNZQYXBSEBEVB5VDI6KGG6WMZFCQQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.816323450 (509h9m35.687406485s) [noise=true , address=145.79.227.6:4259 , piece_id=JI7CPNGZPWJSQOZ4EN6TTHDAGVGUXTJTFKU3BVUR6ZFQUEMXEGGQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.816612370 (509h9m35.687695405s) [noise=true , address=145.79.227.48:4301 , piece_id=BF442XW5DGF53AIULYAYEURTZU5AJHOLBUVYXN3XYY6LWJT7TBOQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.816617861 (509h9m35.687700896s) [noise=true , address=145.79.227.2:4114 , piece_id=SL57BDKDL2IZ6F7YDJY26BZYEYO3MEP7VTPZDY62P7K3GNXZ73QA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817008805 (509h9m35.68809184s) [noise=true , address=145.79.227.50:4375 , piece_id=PZ3HUBDKUYPXBTDNEITVAFA7NKIHU7S5AYL4Y72TYXGCAAMBNEOQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817039053 (509h9m35.688122098s) [noise=true , address=145.79.227.21:4475 , piece_id=3WKC6HE5KF2BQEVOF5I53M4CH2HIY5S2RE3V7SSDJKTTKXYWTWNQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817307664 (509h9m35.688390699s) [noise=true , address=145.79.227.49:4339 , piece_id=Y4CWHTCNEVH7SSZIGNYNX6IBFHNGRY45QIUJJDTNAI5YVHMIZWBQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817451137 (509h9m35.688534173s) [noise=true , address=145.79.227.12:4767 , piece_id=5SPLGOZWUMC5XZWDOJP3SIHAP7JVURGEWN6X22TIKSPTRSC4E3BA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817575524 (509h9m35.688658559s) [noise=true , address=145.79.227.36:4518 , piece_id=QAX3TOLUZGHU3FUTQ7EPZZ3HQLD46EZKFXGLUKNOJ3OSVHL7OXSQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817901034 (509h9m35.688984069s) [noise=true , address=145.79.227.15:4874 , piece_id=EKZSIIAMPHW5EK2QZJWLBTILRAR3FLL2KJFHIOACDOYQIWTQKBDA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.817946000 (509h9m35.689029035s) [noise=true , address=45.91.116.8:4047 , piece_id=F4RHDDCZKUXO3SDSC76PSMQCT2KIT5T2SBUQKSOM2R5AMWIQWJVA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.818155989 (509h9m35.689239024s) [noise=true , address=145.79.227.41:4087 , piece_id=X2LULE7OTX5S62ZZSRED3TVKKTCV4BPYF4XNGTMVS57YIQH3EUAQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.818235951 (509h9m35.689318986s) [noise=true , address=145.79.227.4:4185 , piece_id=7LMP5WY6ZLG2JO5GICZ25JRNMJAL4L2ZADWBKI57MTRVL3Q2TDEA, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:35.818618720 (509h9m35.689701755s) [noise=true , address=45.91.116.10:4084 , piece_id=JY5EP3XRI7JVPU7AH4FY6GQCV52QQ73JHQ73MQVPCG7GHLGLXMWQ, seg_pos={0 0}, upload=446789]: Uploading piece... Dec 26 20:38:37.434486949 (509h9m37.305569994s) [upload=446789]: Next returned nil segment Dec 26 20:38:37.434508218 (509h9m37.305591254s) [upload=446789]: Waiting for error group managing segments... Dec 26 20:38:37.817540224 (509h9m37.688623269s) [noise=true , address=145.79.227.48:4301 , piece_id=BF442XW5DGF53AIULYAYEURTZU5AJHOLBUVYXN3XYY6LWJT7TBOQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.823210662 (509h9m37.694293697s) [noise=true , address=145.79.227.49:4339 , piece_id=Y4CWHTCNEVH7SSZIGNYNX6IBFHNGRY45QIUJJDTNAI5YVHMIZWBQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.833403761 (509h9m37.704486796s) [noise=true , address=145.79.227.15:4874 , piece_id=EKZSIIAMPHW5EK2QZJWLBTILRAR3FLL2KJFHIOACDOYQIWTQKBDA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.833893904 (509h9m37.704976939s) [noise=true , address=145.79.227.50:4375 , piece_id=PZ3HUBDKUYPXBTDNEITVAFA7NKIHU7S5AYL4Y72TYXGCAAMBNEOQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.833929802 (509h9m37.705012837s) [noise=true , address=145.79.227.12:4767 , piece_id=5SPLGOZWUMC5XZWDOJP3SIHAP7JVURGEWN6X22TIKSPTRSC4E3BA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.835668072 (509h9m37.706751117s) [noise=true , address=145.79.227.36:4518 , piece_id=QAX3TOLUZGHU3FUTQ7EPZZ3HQLD46EZKFXGLUKNOJ3OSVHL7OXSQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.836626337 (509h9m37.707709382s) [noise=true , address=145.79.227.21:4475 , piece_id=3WKC6HE5KF2BQEVOF5I53M4CH2HIY5S2RE3V7SSDJKTTKXYWTWNQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:37.850227766 (509h9m37.721310801s) [noise=true , address=145.79.227.38:4978 , piece_id=HWU4OBQOUZBBZQ5PUBHTAJ6KYB4FAFL5LN3SU7RTO4HYUIXIJMSA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:38.112495016 (509h9m37.983578081s) [noise=true , address=45.91.116.8:4047 , piece_id=F4RHDDCZKUXO3SDSC76PSMQCT2KIT5T2SBUQKSOM2R5AMWIQWJVA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:38.144925769 (509h9m38.016008804s) [noise=true , address=77.90.31.13:4627 , piece_id=OHSL7RCQ4V65L3VM2ICIWUOUNZQYXBSEBEVB5VDI6KGG6WMZFCQQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:38.186619000 (509h9m38.057702035s) [noise=true , address=45.91.116.10:4084 , piece_id=JY5EP3XRI7JVPU7AH4FY6GQCV52QQ73JHQ73MQVPCG7GHLGLXMWQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:39.384183618 (509h9m39.255266663s) [noise=true , address=145.79.227.4:4185 , piece_id=7LMP5WY6ZLG2JO5GICZ25JRNMJAL4L2ZADWBKI57MTRVL3Q2TDEA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:39.653749516 (509h9m39.524832552s) [noise=true , address=145.79.227.17:4971 , piece_id=UVNF63SU6VQS2BM7QNJRO7HOW3RZMXOBPXNIZSUI32TRM564J6LQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: <nil> Dec 26 20:38:39.653770356 (509h9m39.524853401s) [seg_pos={0 0}, upload=446789]: Segment reached optimal threshold of 13 pieces. Dec 26 20:38:39.653972170 (509h9m39.525055205s) [noise=true , address=145.79.227.2:4114 , piece_id=SL57BDKDL2IZ6F7YDJY26BZYEYO3MEP7VTPZDY62P7K3GNXZ73QA, seg_pos={0 0}, upload=446789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:46682->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:46682->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46682->145.79.227.2:4114: use of closed network connection Dec 26 20:38:39.653977811 (509h9m39.525060856s) [noise=true , address=145.79.227.2:4114 , piece_id=SL57BDKDL2IZ6F7YDJY26BZYEYO3MEP7VTPZDY62P7K3GNXZ73QA, seg_pos={0 0}, upload=446789]: Long tail cancellation due to optimal threshold Dec 26 20:38:39.654120944 (509h9m39.525203979s) [noise=true , address=145.79.227.41:4087 , piece_id=X2LULE7OTX5S62ZZSRED3TVKKTCV4BPYF4XNGTMVS57YIQH3EUAQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 26 20:38:39.654164707 (509h9m39.525247752s) [noise=true , address=145.79.227.41:4087 , piece_id=X2LULE7OTX5S62ZZSRED3TVKKTCV4BPYF4XNGTMVS57YIQH3EUAQ, seg_pos={0 0}, upload=446789]: Long tail cancellation due to optimal threshold Dec 26 20:38:39.654198221 (509h9m39.525281256s) [noise=true , address=145.79.227.6:4259 , piece_id=JI7CPNGZPWJSQOZ4EN6TTHDAGVGUXTJTFKU3BVUR6ZFQUEMXEGGQ, seg_pos={0 0}, upload=446789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Dec 26 20:38:39.654205044 (509h9m39.525288089s) [noise=true , address=145.79.227.6:4259 , piece_id=JI7CPNGZPWJSQOZ4EN6TTHDAGVGUXTJTFKU3BVUR6ZFQUEMXEGGQ, seg_pos={0 0}, upload=446789]: Long tail cancellation due to optimal threshold Dec 26 20:38:39.654368305 (509h9m39.52545135s) [seg_pos={0 0}, upload=446789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:38:39.654477302 (509h9m39.525560347s) [upload=446789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:38:39.654483324 (509h9m39.525566369s) [upload=446789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:38:39.941361207 (509h9m39.812444242s) [upload=446789]: Done uploading segments... Dec 26 20:38:39.941379090 (509h9m39.812462125s) [upload=446789]: Upload finished. err: <nil> Duration: 30.002571788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..............X...............] duration: 18.370598918s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:39:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 18.370628193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..............X..............] duration: 22.502335577s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:38:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 22.502359803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...] duration: 6.072891508s trace: https://jaeger.dev.storj.io/trace/37e7457c9b66a7a9 FAIL (2025-12-26 20:37:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ) context deadline exceeded ======LOGS====== Dec 26 20:35:59.838675943 (509h6m59.709758978s) [upload=446751]: Starting upload Dec 26 20:35:59.838733052 (509h6m59.709816087s) [upload=446751]: Done starting upload Dec 26 20:35:59.838770424 (509h6m59.709853459s) [upload=446751]: Uploading segments... Dec 26 20:35:59.838922142 (509h6m59.710005177s) [upload=446751]: Got next segment. Inline: false Dec 26 20:35:59.838952480 (509h6m59.710035525s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:35:59.838956568 (509h6m59.710039603s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:36:00.081567129 (509h6m59.952650174s) [seg_pos={0 0}, upload=446751]: Begin upload segment... Dec 26 20:36:00.081625901 (509h6m59.952708936s) [seg_pos={0 0}, upload=446751]: Done begin upload segment. Dec 26 20:36:00.081660326 (509h6m59.952743361s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.081860428 (509h6m59.952943513s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082542836 (509h6m59.953625872s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082960261 (509h6m59.954043296s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083380471 (509h6m59.954463506s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083689089 (509h6m59.954772134s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084001564 (509h6m59.955084609s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084382730 (509h6m59.955465765s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085478167 (509h6m59.956561212s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085488847 (509h6m59.956571882s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.086777120 (509h6m59.957860155s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.088745016 (509h6m59.959828061s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.090076321 (509h6m59.961159356s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.092079636 (509h6m59.963162681s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.093590723 (509h6m59.964673758s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.097499606 (509h6m59.968582651s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:01.791507125 (509h7m1.66259017s) [upload=446751]: Next returned nil segment Dec 26 20:36:01.791523206 (509h7m1.662606241s) [upload=446751]: Waiting for error group managing segments... Dec 26 20:36:02.085454439 (509h7m1.956537474s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090082772 (509h7m1.961165817s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090183653 (509h7m1.961266698s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.102790049 (509h7m1.973873084s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.109495036 (509h7m1.980578071s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.111011504 (509h7m1.982094549s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.115162758 (509h7m1.986245803s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.117677947 (509h7m1.988760992s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.131914075 (509h7m2.00299711s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.357118972 (509h7m2.228202017s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.421069113 (509h7m2.292152158s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.431921228 (509h7m2.303004273s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435118636 (509h7m2.306201681s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435135768 (509h7m2.306218813s) [seg_pos={0 0}, upload=446751]: Segment reached optimal threshold of 13 pieces. Dec 26 20:36:02.435395143 (509h7m2.306478188s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435404941 (509h7m2.306487986s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435618498 (509h7m2.306701553s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435638897 (509h7m2.306721932s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435744398 (509h7m2.306827443s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection Dec 26 20:36:02.435756100 (509h7m2.306839155s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435790806 (509h7m2.306873851s) [seg_pos={0 0}, upload=446751]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:36:02.435890416 (509h7m2.306973461s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:36:02.435897589 (509h7m2.306980634s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:36:02.727681002 (509h7m2.598764057s) [upload=446751]: Done uploading segments... Dec 26 20:36:02.727709406 (509h7m2.598792451s) [upload=446751]: Upload finished. err: <nil> Duration: 6.072920915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..............X.............] duration: 15.160843607s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 15.160878592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX..] duration: 6.255959138s trace: https://jaeger.dev.storj.io/trace/37e7457c9b66a7a9 FAIL (2025-12-26 20:37:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ) context deadline exceeded ======LOGS====== Dec 26 20:35:59.838675943 (509h6m59.709758978s) [upload=446751]: Starting upload Dec 26 20:35:59.838733052 (509h6m59.709816087s) [upload=446751]: Done starting upload Dec 26 20:35:59.838770424 (509h6m59.709853459s) [upload=446751]: Uploading segments... Dec 26 20:35:59.838922142 (509h6m59.710005177s) [upload=446751]: Got next segment. Inline: false Dec 26 20:35:59.838952480 (509h6m59.710035525s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:35:59.838956568 (509h6m59.710039603s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:36:00.081567129 (509h6m59.952650174s) [seg_pos={0 0}, upload=446751]: Begin upload segment... Dec 26 20:36:00.081625901 (509h6m59.952708936s) [seg_pos={0 0}, upload=446751]: Done begin upload segment. Dec 26 20:36:00.081660326 (509h6m59.952743361s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.081860428 (509h6m59.952943513s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082542836 (509h6m59.953625872s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082960261 (509h6m59.954043296s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083380471 (509h6m59.954463506s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083689089 (509h6m59.954772134s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084001564 (509h6m59.955084609s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084382730 (509h6m59.955465765s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085478167 (509h6m59.956561212s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085488847 (509h6m59.956571882s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.086777120 (509h6m59.957860155s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.088745016 (509h6m59.959828061s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.090076321 (509h6m59.961159356s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.092079636 (509h6m59.963162681s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.093590723 (509h6m59.964673758s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.097499606 (509h6m59.968582651s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:01.791507125 (509h7m1.66259017s) [upload=446751]: Next returned nil segment Dec 26 20:36:01.791523206 (509h7m1.662606241s) [upload=446751]: Waiting for error group managing segments... Dec 26 20:36:02.085454439 (509h7m1.956537474s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090082772 (509h7m1.961165817s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090183653 (509h7m1.961266698s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.102790049 (509h7m1.973873084s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.109495036 (509h7m1.980578071s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.111011504 (509h7m1.982094549s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.115162758 (509h7m1.986245803s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.117677947 (509h7m1.988760992s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.131914075 (509h7m2.00299711s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.357118972 (509h7m2.228202017s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.421069113 (509h7m2.292152158s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.431921228 (509h7m2.303004273s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435118636 (509h7m2.306201681s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435135768 (509h7m2.306218813s) [seg_pos={0 0}, upload=446751]: Segment reached optimal threshold of 13 pieces. Dec 26 20:36:02.435395143 (509h7m2.306478188s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435404941 (509h7m2.306487986s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435618498 (509h7m2.306701553s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435638897 (509h7m2.306721932s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435744398 (509h7m2.306827443s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection Dec 26 20:36:02.435756100 (509h7m2.306839155s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435790806 (509h7m2.306873851s) [seg_pos={0 0}, upload=446751]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:36:02.435890416 (509h7m2.306973461s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:36:02.435897589 (509h7m2.306980634s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:36:02.727681002 (509h7m2.598764057s) [upload=446751]: Done uploading segments... Dec 26 20:36:02.727709406 (509h7m2.598792451s) [upload=446751]: Upload finished. err: <nil> Duration: 6.255983846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..............X............] duration: 19.260116247s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:36:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 19.260142538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX.] duration: 7.462102605s trace: https://jaeger.dev.storj.io/trace/37e7457c9b66a7a9 FAIL (2025-12-26 20:36:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ) context deadline exceeded ======LOGS====== Dec 26 20:35:59.838675943 (509h6m59.709758978s) [upload=446751]: Starting upload Dec 26 20:35:59.838733052 (509h6m59.709816087s) [upload=446751]: Done starting upload Dec 26 20:35:59.838770424 (509h6m59.709853459s) [upload=446751]: Uploading segments... Dec 26 20:35:59.838922142 (509h6m59.710005177s) [upload=446751]: Got next segment. Inline: false Dec 26 20:35:59.838952480 (509h6m59.710035525s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:35:59.838956568 (509h6m59.710039603s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:36:00.081567129 (509h6m59.952650174s) [seg_pos={0 0}, upload=446751]: Begin upload segment... Dec 26 20:36:00.081625901 (509h6m59.952708936s) [seg_pos={0 0}, upload=446751]: Done begin upload segment. Dec 26 20:36:00.081660326 (509h6m59.952743361s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.081860428 (509h6m59.952943513s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082542836 (509h6m59.953625872s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082960261 (509h6m59.954043296s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083380471 (509h6m59.954463506s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083689089 (509h6m59.954772134s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084001564 (509h6m59.955084609s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084382730 (509h6m59.955465765s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085478167 (509h6m59.956561212s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085488847 (509h6m59.956571882s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.086777120 (509h6m59.957860155s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.088745016 (509h6m59.959828061s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.090076321 (509h6m59.961159356s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.092079636 (509h6m59.963162681s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.093590723 (509h6m59.964673758s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.097499606 (509h6m59.968582651s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:01.791507125 (509h7m1.66259017s) [upload=446751]: Next returned nil segment Dec 26 20:36:01.791523206 (509h7m1.662606241s) [upload=446751]: Waiting for error group managing segments... Dec 26 20:36:02.085454439 (509h7m1.956537474s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090082772 (509h7m1.961165817s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090183653 (509h7m1.961266698s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.102790049 (509h7m1.973873084s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.109495036 (509h7m1.980578071s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.111011504 (509h7m1.982094549s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.115162758 (509h7m1.986245803s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.117677947 (509h7m1.988760992s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.131914075 (509h7m2.00299711s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.357118972 (509h7m2.228202017s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.421069113 (509h7m2.292152158s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.431921228 (509h7m2.303004273s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435118636 (509h7m2.306201681s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435135768 (509h7m2.306218813s) [seg_pos={0 0}, upload=446751]: Segment reached optimal threshold of 13 pieces. Dec 26 20:36:02.435395143 (509h7m2.306478188s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435404941 (509h7m2.306487986s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435618498 (509h7m2.306701553s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435638897 (509h7m2.306721932s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435744398 (509h7m2.306827443s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection Dec 26 20:36:02.435756100 (509h7m2.306839155s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435790806 (509h7m2.306873851s) [seg_pos={0 0}, upload=446751]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:36:02.435890416 (509h7m2.306973461s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:36:02.435897589 (509h7m2.306980634s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:36:02.727681002 (509h7m2.598764057s) [upload=446751]: Done uploading segments... Dec 26 20:36:02.727709406 (509h7m2.598792451s) [upload=446751]: Upload finished. err: <nil> Duration: 7.462118845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX] duration: 30.00189101s trace: https://jaeger.dev.storj.io/trace/37e7457c9b66a7a9 FAIL (2025-12-26 20:36:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ) context deadline exceeded ======LOGS====== Dec 26 20:35:59.838675943 (509h6m59.709758978s) [upload=446751]: Starting upload Dec 26 20:35:59.838733052 (509h6m59.709816087s) [upload=446751]: Done starting upload Dec 26 20:35:59.838770424 (509h6m59.709853459s) [upload=446751]: Uploading segments... Dec 26 20:35:59.838922142 (509h6m59.710005177s) [upload=446751]: Got next segment. Inline: false Dec 26 20:35:59.838952480 (509h6m59.710035525s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:35:59.838956568 (509h6m59.710039603s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:36:00.081567129 (509h6m59.952650174s) [seg_pos={0 0}, upload=446751]: Begin upload segment... Dec 26 20:36:00.081625901 (509h6m59.952708936s) [seg_pos={0 0}, upload=446751]: Done begin upload segment. Dec 26 20:36:00.081660326 (509h6m59.952743361s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.081860428 (509h6m59.952943513s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082542836 (509h6m59.953625872s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.082960261 (509h6m59.954043296s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083380471 (509h6m59.954463506s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.083689089 (509h6m59.954772134s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084001564 (509h6m59.955084609s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.084382730 (509h6m59.955465765s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085478167 (509h6m59.956561212s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.085488847 (509h6m59.956571882s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.086777120 (509h6m59.957860155s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.088745016 (509h6m59.959828061s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.090076321 (509h6m59.961159356s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.092079636 (509h6m59.963162681s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.093590723 (509h6m59.964673758s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:00.097499606 (509h6m59.968582651s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Uploading piece... Dec 26 20:36:01.791507125 (509h7m1.66259017s) [upload=446751]: Next returned nil segment Dec 26 20:36:01.791523206 (509h7m1.662606241s) [upload=446751]: Waiting for error group managing segments... Dec 26 20:36:02.085454439 (509h7m1.956537474s) [noise=true , address=145.79.227.37:4566 , piece_id=FUAPQYRODLNHGXOWRKWO5CILE2D7Q4FNSMNP2EQ3TXHUYHJLSOBQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090082772 (509h7m1.961165817s) [noise=true , address=145.79.227.48:4302 , piece_id=MAEK46RIIVL34MTSKUWZ52KDM5LFCPVOC2YTRETGJL3JSB3N6HWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.090183653 (509h7m1.961266698s) [noise=true , address=145.79.227.52:4447 , piece_id=ITE4VLCJ4F3VCWQIWOPEJPU4NAU3S36YZAPBPLTY5NU2NZLTOSRA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.102790049 (509h7m1.973873084s) [noise=true , address=145.79.227.45:4230 , piece_id=5WEU2IYWRRUJMEB3G36BSX5FKFXRYDMRHDH7IDFPOIWNH2SDMTCQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.109495036 (509h7m1.980578071s) [noise=true , address=145.79.227.36:4517 , piece_id=MQXWD6PAQ3YSGDJXR5GPW7U2SSD7RKVRAN4YXKRTUVSWHRH4GZ6A, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.111011504 (509h7m1.982094549s) [noise=true , address=145.79.227.46:4603 , piece_id=K42RPWZ6GD5NNNW7DJEKRZTW6S7DHHGLJGEID5Y4HXUQSAGAS4QQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.115162758 (509h7m1.986245803s) [noise=true , address=145.79.227.49:4339 , piece_id=QAAYNMBCVTRWF6RPXSNZ2D2S2APK5PZJAHWFJML22SXREGYCJH3Q, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.117677947 (509h7m1.988760992s) [noise=true , address=145.79.227.13:4803 , piece_id=3KPT32GDZA2HAEUTH2HLVYXRN3GXBL2IWCMKVMRHCLGXGN35QQYQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.131914075 (509h7m2.00299711s) [noise=true , address=145.79.227.11:4732 , piece_id=CRMBCRTHQ3ZMDUMIBGGS52VXDANLRYSPEM2XLLKBLBYBBDR6BNWQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.357118972 (509h7m2.228202017s) [noise=true , address=45.91.116.7:4012 , piece_id=GGVAYTW2VPDFFJLGIPOBQJDGBPRO76EA4NVCXAR3D4TKO26A5ODQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.421069113 (509h7m2.292152158s) [noise=true , address=45.91.116.10:4084 , piece_id=F6KXJJA5ORABUFXX7KCCHCSGZCRCKBFC6TULNO5FZOOV3EHT3MHQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.431921228 (509h7m2.303004273s) [noise=true , address=45.91.116.12:4227 , piece_id=PE2PZXC3NB6O6ZNXC7IBXH3JJCKK2GEVXSB5I3CEKDIWV7XE2TUA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435118636 (509h7m2.306201681s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPUKV73GT756D5VAWUWYSQXETOVTYIWG7LXLZ45DXUKYZLXLSSAQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: <nil> Dec 26 20:36:02.435135768 (509h7m2.306218813s) [seg_pos={0 0}, upload=446751]: Segment reached optimal threshold of 13 pieces. Dec 26 20:36:02.435395143 (509h7m2.306478188s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435404941 (509h7m2.306487986s) [noise=true , address=45.91.116.9:4263 , piece_id=AQ6U7UCPQV2EHUL3TVGHSHVYIN6JJ6WZGILK7ZDPBBGJONJ4UOLQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435618498 (509h7m2.306701553s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 20:36:02.435638897 (509h7m2.306721932s) [noise=true , address=45.91.116.8:4046 , piece_id=H235SERE3EYN4MBEPBYHX5YQOL2VBKC6II3L7OYNRUPK3SPUQLJA, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435744398 (509h7m2.306827443s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37250->145.79.227.2:4113: use of closed network connection Dec 26 20:36:02.435756100 (509h7m2.306839155s) [noise=true , address=145.79.227.2:4113 , piece_id=FITJGRS6JW2QGFEBUT3CJJVJDE2TVMTTGQFQQDTUKSI2ZVG2B2ZQ, seg_pos={0 0}, upload=446751]: Long tail cancellation due to optimal threshold Dec 26 20:36:02.435790806 (509h7m2.306873851s) [seg_pos={0 0}, upload=446751]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:36:02.435890416 (509h7m2.306973461s) [upload=446751]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:36:02.435897589 (509h7m2.306980634s) [upload=446751]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:36:02.727681002 (509h7m2.598764057s) [upload=446751]: Done uploading segments... Dec 26 20:36:02.727709406 (509h7m2.598792451s) [upload=446751]: Upload finished. err: <nil> Duration: 30.00191188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..............X...........] duration: 15.320083489s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:36:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 15.320109969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..............X..........] duration: 35.991692371s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:35:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 35.991705686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XX] duration: 30.005202939s trace: https://jaeger.dev.storj.io/trace/286c4a5f1f27463 FAIL (2025-12-26 20:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: HRFN644X6RUMZOHBXXF7XUOIUOGWKOAIIHPTJLSV46PUWXRQ32SQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3PIS3YIPOTQGIDXCZITPFUXJEYGY4LHI6PVGMC5P7VXBQTQ2OISA) context deadline exceeded ======LOGS====== Dec 26 20:34:45.159870563 (509h5m45.030953598s) [upload=446733]: Starting upload Dec 26 20:34:45.159926049 (509h5m45.031009084s) [upload=446733]: Done starting upload Dec 26 20:34:45.160660247 (509h5m45.031743282s) [upload=446733]: Uploading segments... Dec 26 20:34:45.160707396 (509h5m45.031790442s) [upload=446733]: Got next segment. Inline: false Dec 26 20:34:45.160750518 (509h5m45.031833553s) [upload=446733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:34:45.160755918 (509h5m45.031838954s) [upload=446733]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:34:45.408590608 (509h5m45.279673653s) [seg_pos={0 0}, upload=446733]: Begin upload segment... Dec 26 20:34:45.408632037 (509h5m45.279715072s) [seg_pos={0 0}, upload=446733]: Done begin upload segment. Dec 26 20:34:45.408657364 (509h5m45.279740399s) [noise=true , address=145.79.227.38:4978 , piece_id=LTDFJXAK3MBKEYCCQDCLROG3NPKHOX2DDRTCWWWCHVCIEHNFQR4Q, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.408739541 (509h5m45.279822586s) [noise=true , address=145.79.227.25:4618 , piece_id=PO74G6BRZWTSNSGSMRBSPE27ETENSQLGTLTSPYGU2FGWAHEULMPQ, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.409744936 (509h5m45.280827981s) [noise=true , address=145.79.227.24:4582 , piece_id=KU2K6YSOKWTQ7TWYXFWE3Z6A3G2NB3HDBLYZSBJ7UP3NMLWMVOUA, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.410209731 (509h5m45.281292766s) [noise=true , address=145.79.227.37:4565 , piece_id=KYFEKRVCLUI25QS475VDWCP6FHIHRFZBJFH3BZRPV5YPR7MMTFPA, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.410658936 (509h5m45.281741971s) [noise=true , address=145.79.227.47:4265 , piece_id=SVS34HE74KA2UNYAOP7OQNBK2IR5ZS6I76TDIMTSPQPBNAJ6QINA, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.411018080 (509h5m45.282101115s) [noise=true , address=145.79.227.42:4122 , piece_id=3Z56ZAQPE2ACO557MRVGHOOAGVPIJGGTKTPL74PZEFWG55W6AS7A, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.411387664 (509h5m45.282470699s) [noise=true , address=45.91.116.10:4084 , piece_id=MNFDCLDT7ATFVBEFMHIJKW5HSOI2IAUG7Q5LPDRKVFFJ3X6F3NZQ, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.411748521 (509h5m45.282831556s) [noise=true , address=45.91.116.11:4118 , piece_id=3PIS3YIPOTQGIDXCZITPFUXJEYGY4LHI6PVGMC5P7VXBQTQ2OISA, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.412122603 (509h5m45.283205648s) [noise=true , address=145.79.227.13:4803 , piece_id=EPYL2PRWUVXPF7F7VPHSFW4IISPUI6XXO6JWRBIMI457SWVU6KRA, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.414066965 (509h5m45.28515s) [noise=true , address=77.90.31.13:4629 , piece_id=4VJW4UR63GG5246URII4ESXDA2KDY7RN4GV4TM6ROISC6WJU7C2Q, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.414468590 (509h5m45.285551625s) [noise=true , address=145.79.227.34:4907 , piece_id=7IW3PQ2TBSOXDQPX3ZQC3M2ZQOSPXF2OJT4AWTQIMXF3A5TJXQSQ, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.415113197 (509h5m45.286196232s) [noise=true , address=145.79.227.36:4517 , piece_id=PHEGWCXWBUUR2JJLYY4NSRFPAVOPE5NJU72UP34X5PKQ5SRX7L6Q, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.415491758 (509h5m45.286574793s) [noise=true , address=45.91.116.13:4155 , piece_id=362JV5EDRLG6W5KCZBYKLUWERZ5ON32GSCEMBPQ4Q2ZGNWECDY6Q, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.416226487 (509h5m45.287309532s) [noise=true , address=145.79.227.45:4229 , piece_id=XXIJHDYOOTVVTOGAIRPCY4P3L5MGYPJU5J76FCBPQZJ2RRTFWEDQ, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.416704076 (509h5m45.287787111s) [noise=true , address=145.79.227.49:4337 , piece_id=GTQH43VWJAKOG4HXLHFEAWJYGLTXNUJMT34RF5FS3OO2XJMRMOSQ, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:45.417037832 (509h5m45.288120867s) [noise=true , address=45.91.116.5:4297 , piece_id=HRFN644X6RUMZOHBXXF7XUOIUOGWKOAIIHPTJLSV46PUWXRQ32SQ, seg_pos={0 0}, upload=446733]: Uploading piece... Dec 26 20:34:47.126263557 (509h5m46.997346592s) [upload=446733]: Next returned nil segment Dec 26 20:34:47.126277634 (509h5m46.997360669s) [upload=446733]: Waiting for error group managing segments... Dec 26 20:34:47.406684704 (509h5m47.27776778s) [noise=true , address=145.79.227.49:4337 , piece_id=GTQH43VWJAKOG4HXLHFEAWJYGLTXNUJMT34RF5FS3OO2XJMRMOSQ, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.414794477 (509h5m47.285877522s) [noise=true , address=145.79.227.37:4565 , piece_id=KYFEKRVCLUI25QS475VDWCP6FHIHRFZBJFH3BZRPV5YPR7MMTFPA, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.421898985 (509h5m47.29298202s) [noise=true , address=145.79.227.36:4517 , piece_id=PHEGWCXWBUUR2JJLYY4NSRFPAVOPE5NJU72UP34X5PKQ5SRX7L6Q, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.431195136 (509h5m47.302278181s) [noise=true , address=145.79.227.38:4978 , piece_id=LTDFJXAK3MBKEYCCQDCLROG3NPKHOX2DDRTCWWWCHVCIEHNFQR4Q, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.432119226 (509h5m47.303202262s) [noise=true , address=145.79.227.45:4229 , piece_id=XXIJHDYOOTVVTOGAIRPCY4P3L5MGYPJU5J76FCBPQZJ2RRTFWEDQ, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.434989992 (509h5m47.306073027s) [noise=true , address=145.79.227.42:4122 , piece_id=3Z56ZAQPE2ACO557MRVGHOOAGVPIJGGTKTPL74PZEFWG55W6AS7A, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.454780316 (509h5m47.325863351s) [noise=true , address=145.79.227.13:4803 , piece_id=EPYL2PRWUVXPF7F7VPHSFW4IISPUI6XXO6JWRBIMI457SWVU6KRA, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.455626067 (509h5m47.326709112s) [noise=true , address=145.79.227.47:4265 , piece_id=SVS34HE74KA2UNYAOP7OQNBK2IR5ZS6I76TDIMTSPQPBNAJ6QINA, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.686294457 (509h5m47.557377492s) [noise=true , address=77.90.31.13:4629 , piece_id=4VJW4UR63GG5246URII4ESXDA2KDY7RN4GV4TM6ROISC6WJU7C2Q, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.744221169 (509h5m47.615304214s) [noise=true , address=45.91.116.11:4118 , piece_id=3PIS3YIPOTQGIDXCZITPFUXJEYGY4LHI6PVGMC5P7VXBQTQ2OISA, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:47.770884631 (509h5m47.641967666s) [noise=true , address=45.91.116.10:4084 , piece_id=MNFDCLDT7ATFVBEFMHIJKW5HSOI2IAUG7Q5LPDRKVFFJ3X6F3NZQ, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:48.388015237 (509h5m48.259098272s) [noise=true , address=45.91.116.13:4155 , piece_id=362JV5EDRLG6W5KCZBYKLUWERZ5ON32GSCEMBPQ4Q2ZGNWECDY6Q, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:48.524845965 (509h5m48.395929s) [noise=true , address=45.91.116.5:4297 , piece_id=HRFN644X6RUMZOHBXXF7XUOIUOGWKOAIIHPTJLSV46PUWXRQ32SQ, seg_pos={0 0}, upload=446733]: Done uploading piece. err: <nil> Dec 26 20:34:48.524865863 (509h5m48.395948898s) [seg_pos={0 0}, upload=446733]: Segment reached optimal threshold of 13 pieces. Dec 26 20:34:48.525007122 (509h5m48.396090167s) [noise=true , address=145.79.227.24:4582 , piece_id=KU2K6YSOKWTQ7TWYXFWE3Z6A3G2NB3HDBLYZSBJ7UP3NMLWMVOUA, seg_pos={0 0}, upload=446733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: expected piece hash; context canceled; EOF Dec 26 20:34:48.525012331 (509h5m48.396095366s) [noise=true , address=145.79.227.24:4582 , piece_id=KU2K6YSOKWTQ7TWYXFWE3Z6A3G2NB3HDBLYZSBJ7UP3NMLWMVOUA, seg_pos={0 0}, upload=446733]: Long tail cancellation due to optimal threshold Dec 26 20:34:48.525371566 (509h5m48.396454601s) [noise=true , address=145.79.227.25:4618 , piece_id=PO74G6BRZWTSNSGSMRBSPE27ETENSQLGTLTSPYGU2FGWAHEULMPQ, seg_pos={0 0}, upload=446733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: write tcp 188.166.225.170:44594->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:44594->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44594->145.79.227.25:4618: use of closed network connection Dec 26 20:34:48.525380763 (509h5m48.396463808s) [noise=true , address=145.79.227.25:4618 , piece_id=PO74G6BRZWTSNSGSMRBSPE27ETENSQLGTLTSPYGU2FGWAHEULMPQ, seg_pos={0 0}, upload=446733]: Long tail cancellation due to optimal threshold Dec 26 20:34:48.525390011 (509h5m48.396473046s) [noise=true , address=145.79.227.34:4907 , piece_id=7IW3PQ2TBSOXDQPX3ZQC3M2ZQOSPXF2OJT4AWTQIMXF3A5TJXQSQ, seg_pos={0 0}, upload=446733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 20:34:48.525396252 (509h5m48.396479297s) [noise=true , address=145.79.227.34:4907 , piece_id=7IW3PQ2TBSOXDQPX3ZQC3M2ZQOSPXF2OJT4AWTQIMXF3A5TJXQSQ, seg_pos={0 0}, upload=446733]: Long tail cancellation due to optimal threshold Dec 26 20:34:48.525412854 (509h5m48.39649589s) [seg_pos={0 0}, upload=446733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:34:48.525480173 (509h5m48.396563208s) [upload=446733]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:34:48.525485002 (509h5m48.396568037s) [upload=446733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:34:48.819621145 (509h5m48.69070418s) [upload=446733]: Done uploading segments... Dec 26 20:34:48.819642205 (509h5m48.69072525s) [upload=446733]: Upload finished. err: <nil> Duration: 30.005228686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..............X.........] duration: 25.201744978s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:35:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 25.20177729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..............X........] duration: 27.129779042s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:34:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 27.129793409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..............X.......] duration: 25.418170857s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:34:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 25.418188913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.X] duration: 30.003534379s trace: https://jaeger.dev.storj.io/trace/5fa8d97fa1188780 FAIL (2025-12-26 20:34:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:33:43.226968652 (509h4m43.098051687s) [upload=446718]: Starting upload Dec 26 20:33:43.227024108 (509h4m43.098107143s) [upload=446718]: Done starting upload Dec 26 20:33:43.227118017 (509h4m43.098201052s) [upload=446718]: Uploading segments... Dec 26 20:33:43.227202818 (509h4m43.098285853s) [upload=446718]: Got next segment. Inline: false Dec 26 20:33:43.227231874 (509h4m43.098314909s) [upload=446718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:33:43.227236021 (509h4m43.098319056s) [upload=446718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:33:43.477358606 (509h4m43.348441651s) [seg_pos={0 0}, upload=446718]: Begin upload segment... Dec 26 20:33:43.477406557 (509h4m43.348489602s) [seg_pos={0 0}, upload=446718]: Done begin upload segment. Dec 26 20:33:43.477457344 (509h4m43.348540389s) [noise=true , address=145.79.227.39:4015 , piece_id=6EJMIEWRZUF3NNTBHNTPUQIEOV6TIBIN7K2GMAXK6T334GTVI77A, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.477466661 (509h4m43.348549707s) [noise=true , address=77.90.31.16:4736 , piece_id=R4RB35W26A7AVYERTOBOOL7GFQD6JF373NBIIL7JI5KKLYZ2CK7A, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.477584836 (509h4m43.348667871s) [noise=true , address=45.91.116.8:4048 , piece_id=KXKDWWZJNRHTWWXDCKQAAVHK42XSIG4W77CCTKXR6XNSB5EWMIQQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.478095339 (509h4m43.349178374s) [noise=true , address=45.91.116.7:4012 , piece_id=I7TTPMQBPHL536FJKXLHCW3WCLGMHFARZMGKKXBJDZTQ2DXRTXCA, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.478589529 (509h4m43.349672574s) [noise=true , address=145.79.227.43:4158 , piece_id=JEKKQUH4LWPV6AICTY3ZDJWL6C3MDOL3ECE3XGKZS6JQDGQNY2LQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.478987387 (509h4m43.350070433s) [noise=true , address=145.79.227.48:4301 , piece_id=JCENUAK2T4QEIRL5SFIMH5QURWWJYZ66ZKNHFYVERUUNFZY2RCHQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.479407337 (509h4m43.350490382s) [noise=true , address=45.91.116.5:4298 , piece_id=EZXBYVVNMORFVQNYBEPZEEAGSX2QZ33X2Z6LJCWE4U6HLCTZXDKQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.481154774 (509h4m43.352237819s) [noise=true , address=145.79.227.36:4519 , piece_id=FXSUQPO44CN6O32W7EGJZJVRNRZIOZB5GCPV2L7PZX3L7OQKQHMQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.482891902 (509h4m43.353974937s) [noise=true , address=145.79.227.30:4762 , piece_id=NZNNDTB7GZHMC5URPEJVLI3BOADENEX3OMFQ3YURDM2IXV73VPZA, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.484928971 (509h4m43.356012016s) [noise=true , address=145.79.227.44:4194 , piece_id=3HUVQWN2RLOQRWLAXULKQK2XYQMDECWOREIAGCPCGVL6GGCI6NWA, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.486716595 (509h4m43.35779964s) [noise=true , address=145.79.227.38:4979 , piece_id=7CSLY3KZG5SJ3QNCIDDLO7M47C5OG667DVYH7ASCAEAH67TNJ2AQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.488443102 (509h4m43.359526147s) [noise=true , address=77.90.31.17:4773 , piece_id=O6H2NRU6FMNZR7IIENC7IGOSEQLPQJO47UYLRVP6JCAOHCZRRHOA, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.489309051 (509h4m43.360392096s) [noise=true , address=145.79.227.25:4618 , piece_id=GQU47EYPIHQBVV3HZSFS4X5NYPCKWGRYOK6DDHB6SKIQT756E3WQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.491907069 (509h4m43.362990114s) [noise=true , address=77.90.31.19:4844 , piece_id=MOYOKOLCFJSG54LZROGPRVTTQUZ6G5VQCKSANELCPSPXVSN75HPA, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.495277116 (509h4m43.366360171s) [noise=true , address=145.79.227.50:4374 , piece_id=VSO63CMF5QXEMSI6R7TJFEUAHDGY7RJRM4NHGGA7NAG3EGGFLSSQ, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:43.499609996 (509h4m43.370693041s) [noise=true , address=45.91.116.10:4084 , piece_id=HUZ3QJAFULSQILWNRW6NSK76GNNVBVOSQHO6K6WTGSAAR7H7J4ZA, seg_pos={0 0}, upload=446718]: Uploading piece... Dec 26 20:33:45.190859202 (509h4m45.061942237s) [upload=446718]: Next returned nil segment Dec 26 20:33:45.190871485 (509h4m45.061954521s) [upload=446718]: Waiting for error group managing segments... Dec 26 20:33:45.465405924 (509h4m45.336488959s) [noise=true , address=145.79.227.48:4301 , piece_id=JCENUAK2T4QEIRL5SFIMH5QURWWJYZ66ZKNHFYVERUUNFZY2RCHQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.495545805 (509h4m45.36662884s) [noise=true , address=145.79.227.36:4519 , piece_id=FXSUQPO44CN6O32W7EGJZJVRNRZIOZB5GCPV2L7PZX3L7OQKQHMQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.506107966 (509h4m45.377191001s) [noise=true , address=145.79.227.50:4374 , piece_id=VSO63CMF5QXEMSI6R7TJFEUAHDGY7RJRM4NHGGA7NAG3EGGFLSSQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.522592937 (509h4m45.393675972s) [noise=true , address=145.79.227.38:4979 , piece_id=7CSLY3KZG5SJ3QNCIDDLO7M47C5OG667DVYH7ASCAEAH67TNJ2AQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.584835587 (509h4m45.455918622s) [noise=true , address=145.79.227.25:4618 , piece_id=GQU47EYPIHQBVV3HZSFS4X5NYPCKWGRYOK6DDHB6SKIQT756E3WQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.667237775 (509h4m45.53832081s) [noise=true , address=77.90.31.16:4736 , piece_id=R4RB35W26A7AVYERTOBOOL7GFQD6JF373NBIIL7JI5KKLYZ2CK7A, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.667831756 (509h4m45.538914791s) [noise=true , address=77.90.31.19:4844 , piece_id=MOYOKOLCFJSG54LZROGPRVTTQUZ6G5VQCKSANELCPSPXVSN75HPA, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.698951623 (509h4m45.570034668s) [noise=true , address=77.90.31.17:4773 , piece_id=O6H2NRU6FMNZR7IIENC7IGOSEQLPQJO47UYLRVP6JCAOHCZRRHOA, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.751398230 (509h4m45.622481265s) [noise=true , address=45.91.116.10:4084 , piece_id=HUZ3QJAFULSQILWNRW6NSK76GNNVBVOSQHO6K6WTGSAAR7H7J4ZA, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.767432772 (509h4m45.638515817s) [noise=true , address=45.91.116.7:4012 , piece_id=I7TTPMQBPHL536FJKXLHCW3WCLGMHFARZMGKKXBJDZTQ2DXRTXCA, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:45.835723849 (509h4m45.706806894s) [noise=true , address=45.91.116.5:4298 , piece_id=EZXBYVVNMORFVQNYBEPZEEAGSX2QZ33X2Z6LJCWE4U6HLCTZXDKQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:46.970722191 (509h4m46.841805226s) [noise=true , address=145.79.227.43:4158 , piece_id=JEKKQUH4LWPV6AICTY3ZDJWL6C3MDOL3ECE3XGKZS6JQDGQNY2LQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:47.030538792 (509h4m46.901621827s) [noise=true , address=45.91.116.8:4048 , piece_id=KXKDWWZJNRHTWWXDCKQAAVHK42XSIG4W77CCTKXR6XNSB5EWMIQQ, seg_pos={0 0}, upload=446718]: Done uploading piece. err: <nil> Dec 26 20:33:47.030571484 (509h4m46.901654519s) [seg_pos={0 0}, upload=446718]: Segment reached optimal threshold of 13 pieces. Dec 26 20:33:47.030764232 (509h4m46.901847287s) [noise=true , address=145.79.227.44:4194 , piece_id=3HUVQWN2RLOQRWLAXULKQK2XYQMDECWOREIAGCPCGVL6GGCI6NWA, seg_pos={0 0}, upload=446718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 20:33:47.030772658 (509h4m46.901855693s) [noise=true , address=145.79.227.44:4194 , piece_id=3HUVQWN2RLOQRWLAXULKQK2XYQMDECWOREIAGCPCGVL6GGCI6NWA, seg_pos={0 0}, upload=446718]: Long tail cancellation due to optimal threshold Dec 26 20:33:47.030882687 (509h4m46.901965722s) [noise=true , address=145.79.227.30:4762 , piece_id=NZNNDTB7GZHMC5URPEJVLI3BOADENEX3OMFQ3YURDM2IXV73VPZA, seg_pos={0 0}, upload=446718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 20:33:47.030898137 (509h4m46.901981172s) [noise=true , address=145.79.227.30:4762 , piece_id=NZNNDTB7GZHMC5URPEJVLI3BOADENEX3OMFQ3YURDM2IXV73VPZA, seg_pos={0 0}, upload=446718]: Long tail cancellation due to optimal threshold Dec 26 20:33:47.031052651 (509h4m46.902135686s) [noise=true , address=145.79.227.39:4015 , piece_id=6EJMIEWRZUF3NNTBHNTPUQIEOV6TIBIN7K2GMAXK6T334GTVI77A, seg_pos={0 0}, upload=446718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:59538->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:59538->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59538->145.79.227.39:4015: use of closed network connection Dec 26 20:33:47.031059934 (509h4m46.902142979s) [noise=true , address=145.79.227.39:4015 , piece_id=6EJMIEWRZUF3NNTBHNTPUQIEOV6TIBIN7K2GMAXK6T334GTVI77A, seg_pos={0 0}, upload=446718]: Long tail cancellation due to optimal threshold Dec 26 20:33:47.031075054 (509h4m46.902158099s) [seg_pos={0 0}, upload=446718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:33:47.031157080 (509h4m46.902240115s) [upload=446718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:33:47.031162209 (509h4m46.902245244s) [upload=446718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:33:47.327704614 (509h4m47.198787659s) [upload=446718]: Done uploading segments... Dec 26 20:33:47.327727017 (509h4m47.198810062s) [upload=446718]: Upload finished. err: <nil> Duration: 30.003545941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..............X......] duration: 33.713106139s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:33:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 33.713121209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.] duration: 26.170152153s trace: https://jaeger.dev.storj.io/trace/27e75959268fd1fb FAIL (2025-12-26 20:33:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: DNN4PR3PPXXD2RCXET56LOETGKQUDMD6CNKZ55KBDBJSJPOJNVLQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: GKCV2YUR5Y53HIKQQHQ5DF5NTVZV66FIOXBGQRYBX3RLNTYBC2EQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7EZOTMSXLYP66G3SENQ3RFOE6E7N3P3NNLIGUXN4BCRUXWFYZUYA) context deadline exceeded ======LOGS====== Dec 26 20:32:01.948942244 (509h3m1.820025289s) [upload=446692]: Starting upload Dec 26 20:32:01.949000535 (509h3m1.82008357s) [upload=446692]: Done starting upload Dec 26 20:32:01.949200505 (509h3m1.820283551s) [upload=446692]: Uploading segments... Dec 26 20:32:01.949404915 (509h3m1.82048796s) [upload=446692]: Got next segment. Inline: false Dec 26 20:32:01.949469889 (509h3m1.820552934s) [upload=446692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:32:01.949476802 (509h3m1.820559847s) [upload=446692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:32:02.197003954 (509h3m2.068086999s) [seg_pos={0 0}, upload=446692]: Begin upload segment... Dec 26 20:32:02.197067095 (509h3m2.06815014s) [seg_pos={0 0}, upload=446692]: Done begin upload segment. Dec 26 20:32:02.197094536 (509h3m2.068177581s) [noise=true , address=45.91.116.5:4299 , piece_id=7EZOTMSXLYP66G3SENQ3RFOE6E7N3P3NNLIGUXN4BCRUXWFYZUYA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.197147898 (509h3m2.068230943s) [noise=true , address=145.79.227.51:4410 , piece_id=3E3IKO5CNT4DG6FNNEHBAZZP4HNNBGLXQAUAQ2ICIYFJQMXY3IAQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.197605409 (509h3m2.068688454s) [noise=true , address=145.79.227.12:4767 , piece_id=4AWYUKKCLI2LTMPPVYQHI7QH4ZCL77E7IV62JOKXR65D7HURSITQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.198013797 (509h3m2.069096842s) [noise=true , address=145.79.227.48:4301 , piece_id=G7Y7M3FFO7SQO7ABERSS6YCPNELFKRSK6TD6RXJ6OFSSPADO5BTA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.198553675 (509h3m2.06963672s) [noise=true , address=145.79.227.50:4374 , piece_id=6QE4AJFUNMTP5R35YII7AC4T5CRFXOLJRQOPFR6FBYWXPELHLJGQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.198957294 (509h3m2.070040339s) [noise=true , address=145.79.227.29:4942 , piece_id=SYUAKGRXIF3G32L4BTYXOLEACBIYZWT3Y7TWFILTKI7VEGCGD26Q, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.199323571 (509h3m2.070406616s) [noise=true , address=145.79.227.39:4015 , piece_id=NIAXSR6TS5S7VMQIEAXDEKURUB4QNCRH5OWWJR7GYCHYRUKJWJGA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.199825597 (509h3m2.070908632s) [noise=true , address=145.79.227.40:4049 , piece_id=MXTOJNJDT7P2FT2ALKPHQREOZ4BAIPBFHTPQZNVA7JCYOKR3QZKA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.200342211 (509h3m2.071425256s) [noise=true , address=145.79.227.36:4518 , piece_id=JU4OFTL7PDW425BKPYZS3KRWH7F5IVAQUBZBI5FK5HPUW6WRVWLQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.201290877 (509h3m2.072373922s) [noise=true , address=145.79.227.49:4338 , piece_id=EOQRQNUOPZVZSSJKGWNPM4UYHLFCVD2CCMP2VNS6JEVVMBCZDJIQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.201875891 (509h3m2.072958927s) [noise=true , address=145.79.227.11:4732 , piece_id=MTYQCHNCAOH6YKTSQ2Y3DRJ4JV673JHWAAJFP3NWBPMTIUYYUXOA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.203353184 (509h3m2.074436229s) [noise=true , address=145.79.227.47:4265 , piece_id=7KGEMYTWVRSN4BUQBOQK2Z4VO4EC73IGARPJKERFQUU47LG37PKQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.203712769 (509h3m2.074795814s) [noise=true , address=45.91.116.4:4939 , piece_id=C3WWKTAMVO3PRSBRUGC2IS2W2FXA5SXUZKCAAQNPFDZVSGXKMOCQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.205162319 (509h3m2.076245354s) [noise=true , address=45.91.116.12:4225 , piece_id=DNN4PR3PPXXD2RCXET56LOETGKQUDMD6CNKZ55KBDBJSJPOJNVLQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.206627960 (509h3m2.077711005s) [noise=true , address=45.91.116.14:4189 , piece_id=GKCV2YUR5Y53HIKQQHQ5DF5NTVZV66FIOXBGQRYBX3RLNTYBC2EQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.208047723 (509h3m2.079130768s) [noise=true , address=145.79.227.52:4447 , piece_id=B4OKPZ6XRZF52ZQSJEJSCKUB2V7BAJ7PUDV7ZDX2GFAQ2GE6IZHA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:03.826302155 (509h3m3.6973853s) [upload=446692]: Next returned nil segment Dec 26 20:32:03.826324998 (509h3m3.697408033s) [upload=446692]: Waiting for error group managing segments... Dec 26 20:32:04.213978995 (509h3m4.08506203s) [noise=true , address=145.79.227.39:4015 , piece_id=NIAXSR6TS5S7VMQIEAXDEKURUB4QNCRH5OWWJR7GYCHYRUKJWJGA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.214499897 (509h3m4.085582952s) [noise=true , address=145.79.227.11:4732 , piece_id=MTYQCHNCAOH6YKTSQ2Y3DRJ4JV673JHWAAJFP3NWBPMTIUYYUXOA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.232641842 (509h3m4.103724887s) [noise=true , address=145.79.227.36:4518 , piece_id=JU4OFTL7PDW425BKPYZS3KRWH7F5IVAQUBZBI5FK5HPUW6WRVWLQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.232961731 (509h3m4.104044776s) [noise=true , address=145.79.227.48:4301 , piece_id=G7Y7M3FFO7SQO7ABERSS6YCPNELFKRSK6TD6RXJ6OFSSPADO5BTA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.235278292 (509h3m4.106361337s) [noise=true , address=145.79.227.47:4265 , piece_id=7KGEMYTWVRSN4BUQBOQK2Z4VO4EC73IGARPJKERFQUU47LG37PKQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.239793530 (509h3m4.110876575s) [noise=true , address=145.79.227.50:4374 , piece_id=6QE4AJFUNMTP5R35YII7AC4T5CRFXOLJRQOPFR6FBYWXPELHLJGQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.242143724 (509h3m4.113226769s) [noise=true , address=145.79.227.12:4767 , piece_id=4AWYUKKCLI2LTMPPVYQHI7QH4ZCL77E7IV62JOKXR65D7HURSITQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.248153939 (509h3m4.119236984s) [noise=true , address=145.79.227.29:4942 , piece_id=SYUAKGRXIF3G32L4BTYXOLEACBIYZWT3Y7TWFILTKI7VEGCGD26Q, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.249425882 (509h3m4.120508927s) [noise=true , address=145.79.227.49:4338 , piece_id=EOQRQNUOPZVZSSJKGWNPM4UYHLFCVD2CCMP2VNS6JEVVMBCZDJIQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.267334552 (509h3m4.138417597s) [noise=true , address=145.79.227.52:4447 , piece_id=B4OKPZ6XRZF52ZQSJEJSCKUB2V7BAJ7PUDV7ZDX2GFAQ2GE6IZHA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.471340683 (509h3m4.342423718s) [noise=true , address=45.91.116.14:4189 , piece_id=GKCV2YUR5Y53HIKQQHQ5DF5NTVZV66FIOXBGQRYBX3RLNTYBC2EQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.515932554 (509h3m4.387015599s) [noise=true , address=45.91.116.12:4225 , piece_id=DNN4PR3PPXXD2RCXET56LOETGKQUDMD6CNKZ55KBDBJSJPOJNVLQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.893872974 (509h3m4.764956019s) [noise=true , address=45.91.116.5:4299 , piece_id=7EZOTMSXLYP66G3SENQ3RFOE6E7N3P3NNLIGUXN4BCRUXWFYZUYA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.893898913 (509h3m4.764981968s) [seg_pos={0 0}, upload=446692]: Segment reached optimal threshold of 13 pieces. Dec 26 20:32:04.894142377 (509h3m4.765225422s) [noise=true , address=145.79.227.40:4049 , piece_id=MXTOJNJDT7P2FT2ALKPHQREOZ4BAIPBFHTPQZNVA7JCYOKR3QZKA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 20:32:04.894151966 (509h3m4.765235001s) [noise=true , address=145.79.227.40:4049 , piece_id=MXTOJNJDT7P2FT2ALKPHQREOZ4BAIPBFHTPQZNVA7JCYOKR3QZKA, seg_pos={0 0}, upload=446692]: Long tail cancellation due to optimal threshold Dec 26 20:32:04.894175169 (509h3m4.765258224s) [noise=true , address=45.91.116.4:4939 , piece_id=C3WWKTAMVO3PRSBRUGC2IS2W2FXA5SXUZKCAAQNPFDZVSGXKMOCQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 20:32:04.894188415 (509h3m4.76527146s) [noise=true , address=45.91.116.4:4939 , piece_id=C3WWKTAMVO3PRSBRUGC2IS2W2FXA5SXUZKCAAQNPFDZVSGXKMOCQ, seg_pos={0 0}, upload=446692]: Long tail cancellation due to optimal threshold Dec 26 20:32:04.894515277 (509h3m4.765598322s) [noise=true , address=145.79.227.51:4410 , piece_id=3E3IKO5CNT4DG6FNNEHBAZZP4HNNBGLXQAUAQ2ICIYFJQMXY3IAQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:54560->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:54560->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54560->145.79.227.51:4410: use of closed network connection Dec 26 20:32:04.894526107 (509h3m4.765609152s) [noise=true , address=145.79.227.51:4410 , piece_id=3E3IKO5CNT4DG6FNNEHBAZZP4HNNBGLXQAUAQ2ICIYFJQMXY3IAQ, seg_pos={0 0}, upload=446692]: Long tail cancellation due to optimal threshold Dec 26 20:32:04.894549141 (509h3m4.765632187s) [seg_pos={0 0}, upload=446692]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:32:04.894657357 (509h3m4.765740402s) [upload=446692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:32:04.894664441 (509h3m4.765747486s) [upload=446692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:32:05.189650562 (509h3m5.060733617s) [upload=446692]: Done uploading segments... Dec 26 20:32:05.189677313 (509h3m5.060760358s) [upload=446692]: Upload finished. err: <nil> Duration: 26.170181157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..............X.....] duration: 6.123402528s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:33:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 6.123578122s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)