Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.023400047s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:46:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.023423781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.138486365s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:45:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.13851472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.669632704s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:43:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.669647422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.000676016s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:43:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.000703007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 10.134439332s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:42:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 10.134473308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.8088848s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.808903956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.690128018s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:41:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.690157486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.006956437s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:40:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.006978879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.795802235s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:40:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.795833504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 12.668883017s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:39:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 12.668916081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.754166613s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:38:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.754187673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.984053807s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:37:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.984078836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.934355757s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:36:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.934386655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.747077747s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:36:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.747112142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.933142754s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:35:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.933182701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.707051538s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.707110851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 21.320683347s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:34:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 21.320713276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.061864141s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:33:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.06189044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.500925885s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:33:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.500959981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.877145365s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:32:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.877279711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 7.09301765s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:31:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 7.093039831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.043501406s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:30:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.043530642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.957391547s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.957429609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.111817869s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:28:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.111846483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.998660138s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:27:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.998688521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.004068639s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:27:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.004148681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.813347784s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.813374745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............................................X.............] duration: 6.021510642s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:25:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.021536482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.............................................X............] duration: 5.680065854s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:24:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.680090891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XX.............................................X...........] duration: 5.746132284s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:24:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.746160559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XX.............................................X..........] duration: 10.45656677s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 10.456594001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..XX.............................................X.........] duration: 5.9801106s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:22:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.980129988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..XX.............................................X........] duration: 6.160898702s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:22:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.160932427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..XX.............................................X.......] duration: 5.984895762s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:21:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.984922052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X..XX.............................................X......] duration: 5.724844562s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.724884479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X..XX.............................................X.....] duration: 6.061982479s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:19:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.062011324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X..X..XX.............................................X....] duration: 6.001567068s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:19:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.001594712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.X..X..XX.............................................X...] duration: 5.688947539s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:18:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 5.688977086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.X..X..XX.............................................X..] duration: 6.131016101s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:17:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 6.13104164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.X..X..XX.............................................X.] duration: 22.608963878s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:17:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 22.608990028s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X.X..X..XX.............................................X] duration: 30.0033822s trace: https://jaeger.dev.storj.io/trace/454695b11361f7db FAIL (2025-12-22 18:16:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ) context deadline exceeded ======LOGS====== Dec 22 18:15:49.400152692 (410h46m49.271235727s) [upload=360636]: Starting upload Dec 22 18:15:49.400229548 (410h46m49.271312584s) [upload=360636]: Done starting upload Dec 22 18:15:49.400306094 (410h46m49.271389129s) [upload=360636]: Uploading segments... Dec 22 18:15:49.400555229 (410h46m49.271638264s) [upload=360636]: Got next segment. Inline: false Dec 22 18:15:49.400603811 (410h46m49.271686856s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 18:15:49.400611275 (410h46m49.27169432s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 18:15:49.642016540 (410h46m49.513099585s) [seg_pos={0 0}, upload=360636]: Begin upload segment... Dec 22 18:15:49.642063931 (410h46m49.513146966s) [seg_pos={0 0}, upload=360636]: Done begin upload segment. Dec 22 18:15:49.642097294 (410h46m49.513180329s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642104388 (410h46m49.513187433s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642558733 (410h46m49.513641778s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.642944818 (410h46m49.514027853s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643284054 (410h46m49.514367089s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643569297 (410h46m49.514652332s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.643846876 (410h46m49.514929911s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644137038 (410h46m49.515220073s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644421530 (410h46m49.515504565s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.644706733 (410h46m49.515789768s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645115392 (410h46m49.516198427s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645141420 (410h46m49.516224455s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645513359 (410h46m49.516596404s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.645774476 (410h46m49.516857521s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646082372 (410h46m49.517165407s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:49.646368778 (410h46m49.517451823s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Uploading piece... Dec 22 18:15:51.349005768 (410h46m51.220088803s) [upload=360636]: Next returned nil segment Dec 22 18:15:51.349018782 (410h46m51.220101817s) [upload=360636]: Waiting for error group managing segments... Dec 22 18:15:51.628945218 (410h46m51.500028253s) [noise=true , address=145.79.227.50:4373 , piece_id=6D6ZKIN7HYUKVRMQQ7HYKGSPVA5XWQJBUG5MD2IFIVEHGOS5DW3A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632484768 (410h46m51.503567803s) [noise=true , address=145.79.227.7:4294 , piece_id=HBDS4F6IEWWE54KKHNMYXJZOQR4567GT7FJYWM43TXOBPMNABX3Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.632848882 (410h46m51.503931917s) [noise=true , address=145.79.227.37:4566 , piece_id=4CG3CKHTC2MILDNTGYQGDPW54Z473R6NZVBVCT2K7VOXXIWE3PZQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.635614919 (410h46m51.506697964s) [noise=true , address=145.79.227.42:4122 , piece_id=JXXL7F2JLGU55Z76HPAY2INNP32HR3KGK2YIDPKS2EZ5IJGBFZMQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.636534389 (410h46m51.507617424s) [noise=true , address=145.79.227.48:4302 , piece_id=TD3TOWJVT222DOEWDDBD3LR2TRUYSKQOB64JGA3LX5DG2N7DD7SA, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.641309752 (410h46m51.512392787s) [noise=true , address=145.79.227.26:4653 , piece_id=XOU4SUE5SE2VQ6ZDTDVJTKZXGJ6SSMCFIW3HGSW4MUDZEN6RZCUQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.644248258 (410h46m51.515331303s) [noise=true , address=145.79.227.49:4339 , piece_id=5OTE52Y3RRFPXJLUR5HYAUW5MXBBJB6U4VN2WJI5KOTCV572Z6BQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646028047 (410h46m51.517111092s) [noise=true , address=145.79.227.47:4265 , piece_id=EA43Q7J6ACTIMIL2KQDLHTZLVXK6DTF5W7L7RTK37WEQTJW5PMCQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.646161101 (410h46m51.517244156s) [noise=true , address=145.79.227.52:4445 , piece_id=ODYOLZIBZ2KWEKPZWIPZHPJSNWRXY4T56I4OGPQETKY7ALKRSQJQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.827938745 (410h46m51.69902178s) [noise=true , address=77.90.31.16:4736 , piece_id=DHFC43AZK5NBBWWKZFLURBYUT262OHFOTKBK6K3L2J7NSU523RHQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.843200315 (410h46m51.71428335s) [noise=true , address=77.90.31.12:4590 , piece_id=KAT24J6J74IZJBBASILBID3IHFX33763M5PELEWQ3Q5JC7NS4DYQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.852718740 (410h46m51.723801775s) [noise=true , address=77.90.31.14:4661 , piece_id=YGZ3234WTY3RUCLKON7R7CJ555W37WLQSOOKWBPLHFOXJE6ML63A, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887896113 (410h46m51.758979159s) [noise=true , address=45.91.116.5:4300 , piece_id=Y3EVNB5PERFEBSQRQUOY677JKXJQ3TPRBZO4MN3W6TICS7O47MAQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: <nil> Dec 22 18:15:51.887918706 (410h46m51.759001751s) [seg_pos={0 0}, upload=360636]: Segment reached optimal threshold of 13 pieces. Dec 22 18:15:51.888254736 (410h46m51.759337781s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888264544 (410h46m51.759347579s) [noise=true , address=45.91.116.7:4011 , piece_id=TWMV5EEDW4SOKD2UWKBYBD5XSFR32TUD3HWBNLBGLO3ELDWQCS4Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888549338 (410h46m51.759632373s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888569075 (410h46m51.75965211s) [noise=true , address=45.91.116.11:4120 , piece_id=UIVZD2MVNCPRA7INA4V3BEIJGJD4PGHPFJIGZQUNU6EWZMHALK7Q, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888821535 (410h46m51.75990457s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 22 18:15:51.888828349 (410h46m51.759911394s) [noise=true , address=45.91.116.6:4975 , piece_id=QC5UKJG7H6DBZRMI6KHLG5HDXV5UCEOBBTP6L5ZBACSFYV5QI5CQ, seg_pos={0 0}, upload=360636]: Long tail cancellation due to optimal threshold Dec 22 18:15:51.888848897 (410h46m51.759931942s) [seg_pos={0 0}, upload=360636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 18:15:51.889079136 (410h46m51.760162171s) [upload=360636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 18:15:51.889087171 (410h46m51.760170206s) [upload=360636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 18:15:52.183033714 (410h46m52.054116749s) [upload=360636]: Done uploading segments... Dec 22 18:15:52.183050115 (410h46m52.05413315s) [upload=360636]: Upload finished. err: <nil> Duration: 30.003403701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...X.X..X..XX.............................................] duration: 8.407179325s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:14:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 8.407205535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...X.X..X..XX............................................] duration: 5.97823969s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:13:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.978267743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...X.X..X..XX...........................................] duration: 5.937782902s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:13:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.937817175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...X.X..X..XX..........................................] duration: 5.940745163s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:12:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.940769148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....X...X.X..X..XX.........................................] duration: 5.996237412s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:11:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.996264041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX....X...X.X..X..XX........................................] duration: 6.958585228s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:11:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 6.958652996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX....X...X.X..X..XX.......................................] duration: 7.653870153s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 7.653898637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX....X...X.X..X..XX......................................] duration: 7.369524941s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:09:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 7.369554779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..XX....X...X.X..X..XX.....................................] duration: 5.971975813s trace: https://jaeger.dev.storj.io/trace/11bad95976db3bb5 FAIL (2025-12-22 18:08:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA) context deadline exceeded ======LOGS====== Dec 22 17:38:35.883524653 (410h9m35.754607698s) [upload=360083]: Starting upload Dec 22 17:38:35.883589467 (410h9m35.754672502s) [upload=360083]: Done starting upload Dec 22 17:38:35.884099608 (410h9m35.755182643s) [upload=360083]: Uploading segments... Dec 22 17:38:35.884144413 (410h9m35.755227458s) [upload=360083]: Got next segment. Inline: false Dec 22 17:38:35.884186804 (410h9m35.755269839s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 17:38:35.884192275 (410h9m35.75527531s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 17:38:36.132982263 (410h9m36.004065308s) [seg_pos={0 0}, upload=360083]: Begin upload segment... Dec 22 17:38:36.133045324 (410h9m36.004128369s) [seg_pos={0 0}, upload=360083]: Done begin upload segment. Dec 22 17:38:36.133081382 (410h9m36.004164427s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.137981683 (410h9m36.009064728s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.140116358 (410h9m36.011199393s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142408623 (410h9m36.013491658s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.142434782 (410h9m36.013517827s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.143637162 (410h9m36.014720197s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144422868 (410h9m36.015505914s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.144823000 (410h9m36.015906035s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.145994491 (410h9m36.017077526s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.146457021 (410h9m36.017540067s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.147245273 (410h9m36.018328318s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148254353 (410h9m36.019337388s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.148272659 (410h9m36.019355694s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.149675469 (410h9m36.020758505s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.150150154 (410h9m36.021233199s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:36.151782192 (410h9m36.022865227s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Uploading piece... Dec 22 17:38:37.276276889 (410h9m37.147359925s) [noise=true , address=77.90.31.21:4877 , piece_id=XZ6ZDEDG3FE7CTER7KS5KEM7R3QUENE5CDR3DFJ5WCKJUGZY3KUA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 22 17:38:38.015936327 (410h9m37.887019372s) [upload=360083]: Next returned nil segment Dec 22 17:38:38.015949371 (410h9m37.887032406s) [upload=360083]: Waiting for error group managing segments... Dec 22 17:38:38.363457204 (410h9m38.234540249s) [noise=true , address=45.91.116.10:4082 , piece_id=WQNYIGXB242L6CI4FFBXK6KZEKL3GTQNAZVKCHU43CHP7VY2MFHA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.367439257 (410h9m38.238522292s) [noise=true , address=45.91.116.9:4262 , piece_id=O7RPEE3O5QMZZZ3KKYUBHJ2H6R3HNGDOKRM2UANAJ73U3B2APIMA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.432943829 (410h9m38.304026874s) [noise=true , address=45.91.116.11:4117 , piece_id=N4SBAXDNIJK7I7OPIBO35NTIVUGV6XFS4XEWFMW3U74SUS72O6QA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.747311478 (410h9m38.618394513s) [noise=true , address=45.91.116.8:4048 , piece_id=L3J522CNDSOEDKQGEJTY3WVCHGQLAI34PN7Z2D64R6GLCJ6XUGVQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.953237456 (410h9m38.824320491s) [noise=true , address=145.79.227.49:4337 , piece_id=K6NRSHOJVCSF7SCVB6WEG2CAQLYRYFPKBZMP4OAWPUH26E2WHGQA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.970804947 (410h9m38.841887982s) [noise=true , address=145.79.227.51:4409 , piece_id=TZIQ2AD6HS42EXAEEKL6OGTJI3HR2TXHOU572F2X6QACT4VQVM3Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:38.981963225 (410h9m38.85304627s) [noise=true , address=145.79.227.19:4078 , piece_id=BOE4LXEXQTDAHDP2NBOHJSORDVXS4SH5GCMM7AKPXHRKBZTCO5LA, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.014542351 (410h9m38.885625386s) [noise=true , address=145.79.227.15:4875 , piece_id=EGUVHN72QQCPXT73TDZP44HWDOZZH2CNTY6KS6T62VHYYKX6W62Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109157525 (410h9m38.98024057s) [noise=true , address=145.79.227.22:4509 , piece_id=YVEF7YFWVCAW4272TVQMR6T7GMCTVXAZA2LS6EYDQM3RUGBK3Y6Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.109642829 (410h9m38.980725874s) [noise=true , address=145.79.227.45:4231 , piece_id=CKSXZP3WE4QZSXOPUW76R5PZ6PIFVXTBSLYBMFRUDGHPKQAJRN3A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.110145516 (410h9m38.981228551s) [noise=true , address=145.79.227.37:4567 , piece_id=NQMBR226SSLO65FOWCQJOSC3IBSTCAAKHPCJEXLKPB3B2B5O6D2A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.115136960 (410h9m38.986219995s) [noise=true , address=145.79.227.36:4517 , piece_id=VTPQHLF3C4IN7WZZQG4Z4WL52GLKTHQ6T4TMLEVH5FRZAYRL4T7Q, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116650232 (410h9m38.987733277s) [noise=true , address=145.79.227.47:4267 , piece_id=RZTFFZWK5OJHMR77FZAEGZLWMUNOQEXYO5E3VAHMSRJD4MWRFEDQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: <nil> Dec 22 17:38:39.116658828 (410h9m38.987741863s) [seg_pos={0 0}, upload=360083]: Segment reached optimal threshold of 13 pieces. Dec 22 17:38:39.116858579 (410h9m38.987941624s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.116871964 (410h9m38.987955029s) [noise=true , address=145.79.227.26:4655 , piece_id=KXRMBRU5LPFN5C77EHYJS6PNSFBNONZA47APMGGOCDSHDESA4VTQ, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117115498 (410h9m38.988198533s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 17:38:39.117121809 (410h9m38.988204845s) [noise=true , address=145.79.227.42:4123 , piece_id=KITU5CDJUVR37ZO42V7FDOQLVE5MRKZF5VA26LL7A2H6TAFPKZ5A, seg_pos={0 0}, upload=360083]: Long tail cancellation due to optimal threshold Dec 22 17:38:39.117146597 (410h9m38.988229642s) [seg_pos={0 0}, upload=360083]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 17:38:39.117211170 (410h9m38.988294205s) [upload=360083]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 17:38:39.117216590 (410h9m38.988299625s) [upload=360083]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 17:38:39.409860703 (410h9m39.280943759s) [upload=360083]: Done uploading segments... Dec 22 17:38:39.409893245 (410h9m39.280976291s) [upload=360083]: Upload finished. err: <nil> Duration: 5.971999997s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)