Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.97958155s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:49:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.97960797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.751999757s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:48:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.752026648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.776702865s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.776734796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.646109884s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:46:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.646134661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.705012633s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:45:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.705036859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.778336965s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:44:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.778370047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.749599538s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:43:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.749659362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.673577226s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:43:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.673609718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.696999954s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.697048877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.646308506s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:41:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.646355316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.935488725s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:40:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.935517421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.652299195s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:39:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.652334201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.689858816s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:38:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.689924591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.747273385s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:38:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.747331004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.960352422s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:37:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.960381587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.671861896s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:37:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.671885441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.84591533s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:36:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.845957572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.744985722s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:35:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.74501134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.716030032s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:34:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.716060219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.690588356s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:33:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.690614916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.800524977s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:32:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.800538372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.642601767s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:32:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.642632526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.681263532s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:30:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.681292869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.063761263s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:30:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 6.063835185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 17.102507937s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:30:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 17.102537465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 9.618612513s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 9.618652138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.711022968s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:27:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.711053216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.613641535s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:26:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.613667423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.784635459s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:26:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.784663464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 30.003545729s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:25:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 30.003573182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.729615151s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:24:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.72964671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.633563293s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:22:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.633594714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.799900642s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:21:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.799928013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.617406197s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:21:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.617438831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.727705339s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.727742618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.798029367s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.7980686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.645912729s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.645943597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.691210053s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:18:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.691242725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.744378318s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:18:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.744396532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.766633635s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.766663392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.742890671s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.742923435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.667275177s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:16:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.667335103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.743029445s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.743056867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.626389683s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:13:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.626416034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.755877623s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:13:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.756002711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.733924077s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:12:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.733955306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.644267655s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:11:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.644302612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.667910788s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:10:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.667944032s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.603633902s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:09:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.603668747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.760674835s trace: https://jaeger.dev.storj.io/trace/2557f4b7cc63834c FAIL (2025-12-04 11:09:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 10:39:29.282795885 (17h44m5.754006615s) [upload=15593]: Starting upload Dec 4 10:39:29.282878002 (17h44m5.754088742s) [upload=15593]: Done starting upload Dec 4 10:39:29.282934770 (17h44m5.754145509s) [upload=15593]: Uploading segments... Dec 4 10:39:29.283133248 (17h44m5.754343977s) [upload=15593]: Got next segment. Inline: false Dec 4 10:39:29.283176009 (17h44m5.754386738s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 10:39:29.283180699 (17h44m5.754391428s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 10:39:29.534205445 (17h44m6.005416185s) [seg_pos={0 0}, upload=15593]: Begin upload segment... Dec 4 10:39:29.534271311 (17h44m6.00548206s) [seg_pos={0 0}, upload=15593]: Done begin upload segment. Dec 4 10:39:29.534330192 (17h44m6.005540931s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534340091 (17h44m6.00555083s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.534849431 (17h44m6.00606017s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535302925 (17h44m6.006513654s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.535924979 (17h44m6.007135708s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.536820725 (17h44m6.008031464s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.537557258 (17h44m6.008767987s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.539895570 (17h44m6.011106309s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.542161015 (17h44m6.013371744s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.545354665 (17h44m6.016565404s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.548180727 (17h44m6.019391456s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.549966167 (17h44m6.021176906s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.550245318 (17h44m6.021456067s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.552149004 (17h44m6.023359743s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.553130352 (17h44m6.024341091s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:29.556980604 (17h44m6.028191333s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Uploading piece... Dec 4 10:39:31.240503083 (17h44m7.711713812s) [upload=15593]: Next returned nil segment Dec 4 10:39:31.240522991 (17h44m7.711733721s) [upload=15593]: Waiting for error group managing segments... Dec 4 10:39:31.521922248 (17h44m7.993132977s) [noise=true , address=145.79.227.14:4839 , piece_id=QNMKPX4DQ63HJK24CVS7QQ5OM2QPFU2UE2X2CTQP36LVVSJIEIHQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.533429560 (17h44m8.004640299s) [noise=true , address=145.79.227.32:4836 , piece_id=6RRD6BF2YCFQ37VV57XFMR7IWPPSJFJ62CFOIGSN6IYGXF6O43MQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.534955846 (17h44m8.006166575s) [noise=true , address=145.79.227.42:4121 , piece_id=VHOT7ZQDOLMWQJFOXD6ALIS3OV3TYIRS6HADXLRL536JJQ32DFKQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.538352605 (17h44m8.009563334s) [noise=true , address=145.79.227.9:4366 , piece_id=NZOF36Z7RUGMJXW6SU6B27ZYSDEHSVM7GFLZFRVZA442WUZ3HUSQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.539890453 (17h44m8.011101182s) [noise=true , address=145.79.227.10:4403 , piece_id=YSESRTRBMKYR4CXBKLD4DIM2KMSH24A2DVZSVRAF2S6CQPJKZTKA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546061855 (17h44m8.017272584s) [noise=true , address=145.79.227.26:4653 , piece_id=MA5N5SBKO5U42TGJZAHGQT46DHGTSZR3FHP2EI6BXRZF6HLYJYHA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546357678 (17h44m8.017568407s) [noise=true , address=145.79.227.21:4473 , piece_id=PCDS53YNLYZGK7II4JPZJ3O2NNMAFRIEYHQLKTEFA4TGGRV7EVLA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.546598537 (17h44m8.017809276s) [noise=true , address=145.79.227.22:4509 , piece_id=2BYQMI35YWTJGPXBNPRPW6HGVPB3QO4B2X3RDSJ33NJXJTYI6QIQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.550505307 (17h44m8.021716036s) [noise=true , address=145.79.227.34:4907 , piece_id=5Q6IIMJ63SBRHW3R4NRUOGT674LLWEBIPOJ7KO66YCEALZIVD57Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.551476025 (17h44m8.022686754s) [noise=true , address=145.79.227.25:4618 , piece_id=EHE2ECGJOEUYE7CGUHJEWDZYZHDFJXOGFR5LSR7J6H6QAGY6UBJQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.558499569 (17h44m8.029710298s) [noise=true , address=145.79.227.31:4797 , piece_id=FRRF5MWSK4PSLDBMDDOJLK2F3VA5ELT4CGPR2XYJURG5VL3L65DA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.568343785 (17h44m8.039554514s) [noise=true , address=145.79.227.3:4151 , piece_id=UK23UYUQ635R4KNSLS5ONCQIGVTFIO6EOU634P6DG2V34JZ26A3Q, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594795905 (17h44m8.066006644s) [noise=true , address=145.79.227.24:4582 , piece_id=NIC643EZNCPM4Z3OQV7IUEQZGIOAAF3ISCZVB5NRQWWHJ3RNK45A, seg_pos={0 0}, upload=15593]: Done uploading piece. err: <nil> Dec 4 10:39:31.594819370 (17h44m8.066030109s) [seg_pos={0 0}, upload=15593]: Segment reached optimal threshold of 13 pieces. Dec 4 10:39:31.595040641 (17h44m8.06625138s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595049238 (17h44m8.066259977s) [noise=true , address=77.90.31.17:4771 , piece_id=2QNV4LOR2GP4BCJVOOV5X73IYEANCIZ5VBAZS5YRQBZUAOUUVRPA, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595256372 (17h44m8.066467102s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 4 10:39:31.595267714 (17h44m8.066478453s) [noise=true , address=77.90.31.10:4517 , piece_id=LQSIP6QOULSRMNLWMVKG4PVPBK2SZXEIXS5QAWE36MFUSX2SWIYQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595282933 (17h44m8.066493672s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41852->77.90.31.16:4734: use of closed network connection Dec 4 10:39:31.595292471 (17h44m8.06650321s) [noise=true , address=77.90.31.16:4734 , piece_id=ZJKCZFCPZ7ENTECFNP3JZNHCYMPUWMZ2WDGCGN6MGMQEKYNJBRCQ, seg_pos={0 0}, upload=15593]: Long tail cancellation due to optimal threshold Dec 4 10:39:31.595310886 (17h44m8.066521615s) [seg_pos={0 0}, upload=15593]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 10:39:31.595386740 (17h44m8.066597489s) [upload=15593]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 10:39:31.595392521 (17h44m8.06660326s) [upload=15593]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 10:39:31.890512025 (17h44m8.361722764s) [upload=15593]: Done uploading segments... Dec 4 10:39:31.890533596 (17h44m8.361744335s) [upload=15593]: Upload finished. err: <nil> Duration: 5.760705053s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)