Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.575976141s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:57:28 UTC) orbiter warning: context deadline exceeded Duration: 3.576002571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.............XX.....X.X......X.X..XX.X....X..XXX..XX.....X..] duration: 16.648459815s trace: https://jaeger.dev.storj.io/trace/1aa3b31e9fcd916c FAIL (2025-12-30 01:57:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ) context deadline exceeded ======LOGS====== 01:55:26.264693797 (586h26m26.135776842s) [upload=514094]: Starting upload Dec 30 01:55:26.264768260 (586h26m26.135851295s) [upload=514094]: Done starting upload Dec 30 01:55:26.264868770 (586h26m26.135951826s) [upload=514094]: Uploading segments... Dec 30 01:55:26.265054515 (586h26m26.13613755s) [upload=514094]: Got next segment. Inline: false Dec 30 01:55:26.265090924 (586h26m26.136173959s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:55:26.265096034 (586h26m26.136179069s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:55:26.508102931 (586h26m26.379185966s) [seg_pos={0 0}, upload=514094]: Begin upload segment... Dec 30 01:55:26.508145091 (586h26m26.379228126s) [seg_pos={0 0}, upload=514094]: Done begin upload segment. Dec 30 01:55:26.508171071 (586h26m26.379254106s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508250702 (586h26m26.379333767s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508748820 (586h26m26.379831855s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509099648 (586h26m26.380182683s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509393218 (586h26m26.380476253s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509679443 (586h26m26.380762478s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510089403 (586h26m26.381172448s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510402118 (586h26m26.381485163s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510723771 (586h26m26.381806806s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.511615949 (586h26m26.382699005s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.513000306 (586h26m26.384083351s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.514708609 (586h26m26.385791654s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.516561266 (586h26m26.387644301s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.518322920 (586h26m26.389405955s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.519685224 (586h26m26.390768269s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.520165879 (586h26m26.391248914s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:28.113351882 (586h26m27.984434917s) [upload=514094]: Next returned nil segment Dec 30 01:55:28.113366409 (586h26m27.984449444s) [upload=514094]: Waiting for error group managing segments... Dec 30 01:55:28.489714050 (586h26m28.360797085s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.495852478 (586h26m28.366935513s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.496659264 (586h26m28.367742299s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.499030169 (586h26m28.370113204s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.500746907 (586h26m28.371829982s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.504146861 (586h26m28.375229906s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512015273 (586h26m28.383098309s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512474377 (586h26m28.383557413s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.516709301 (586h26m28.387792346s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.519143556 (586h26m28.390226611s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.522158937 (586h26m28.393241972s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.717343337 (586h26m28.588426402s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786975082 (586h26m28.658058117s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786995331 (586h26m28.658078376s) [seg_pos={0 0}, upload=514094]: Segment reached optimal threshold of 13 pieces. Dec 30 01:55:28.787182006 (586h26m28.658265051s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787188408 (586h26m28.658271453s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787225378 (586h26m28.658308423s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection Dec 30 01:55:28.787234876 (586h26m28.658317921s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787382778 (586h26m28.658465823s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787388960 (586h26m28.658471995s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787402736 (586h26m28.658485781s) [seg_pos={0 0}, upload=514094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:55:28.787461819 (586h26m28.658544864s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:55:28.787467429 (586h26m28.658550464s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:55:29.079428950 (586h26m28.950511995s) [upload=514094]: Done uploading segments... Dec 30 01:55:29.079451724 (586h26m28.950534769s) [upload=514094]: Upload finished. err: <nil> Duration: 16.648484683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.454900988s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:56:57 UTC) orbiter warning: context deadline exceeded Duration: 3.454934261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..............XX.....X.X......X.X..XX.X....X..XXX..XX.....X.] duration: 13.06857636s trace: https://jaeger.dev.storj.io/trace/1aa3b31e9fcd916c FAIL (2025-12-30 01:56:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ) context deadline exceeded ======LOGS====== 01:55:26.264693797 (586h26m26.135776842s) [upload=514094]: Starting upload Dec 30 01:55:26.264768260 (586h26m26.135851295s) [upload=514094]: Done starting upload Dec 30 01:55:26.264868770 (586h26m26.135951826s) [upload=514094]: Uploading segments... Dec 30 01:55:26.265054515 (586h26m26.13613755s) [upload=514094]: Got next segment. Inline: false Dec 30 01:55:26.265090924 (586h26m26.136173959s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:55:26.265096034 (586h26m26.136179069s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:55:26.508102931 (586h26m26.379185966s) [seg_pos={0 0}, upload=514094]: Begin upload segment... Dec 30 01:55:26.508145091 (586h26m26.379228126s) [seg_pos={0 0}, upload=514094]: Done begin upload segment. Dec 30 01:55:26.508171071 (586h26m26.379254106s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508250702 (586h26m26.379333767s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508748820 (586h26m26.379831855s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509099648 (586h26m26.380182683s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509393218 (586h26m26.380476253s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509679443 (586h26m26.380762478s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510089403 (586h26m26.381172448s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510402118 (586h26m26.381485163s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510723771 (586h26m26.381806806s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.511615949 (586h26m26.382699005s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.513000306 (586h26m26.384083351s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.514708609 (586h26m26.385791654s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.516561266 (586h26m26.387644301s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.518322920 (586h26m26.389405955s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.519685224 (586h26m26.390768269s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.520165879 (586h26m26.391248914s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:28.113351882 (586h26m27.984434917s) [upload=514094]: Next returned nil segment Dec 30 01:55:28.113366409 (586h26m27.984449444s) [upload=514094]: Waiting for error group managing segments... Dec 30 01:55:28.489714050 (586h26m28.360797085s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.495852478 (586h26m28.366935513s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.496659264 (586h26m28.367742299s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.499030169 (586h26m28.370113204s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.500746907 (586h26m28.371829982s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.504146861 (586h26m28.375229906s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512015273 (586h26m28.383098309s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512474377 (586h26m28.383557413s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.516709301 (586h26m28.387792346s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.519143556 (586h26m28.390226611s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.522158937 (586h26m28.393241972s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.717343337 (586h26m28.588426402s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786975082 (586h26m28.658058117s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786995331 (586h26m28.658078376s) [seg_pos={0 0}, upload=514094]: Segment reached optimal threshold of 13 pieces. Dec 30 01:55:28.787182006 (586h26m28.658265051s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787188408 (586h26m28.658271453s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787225378 (586h26m28.658308423s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection Dec 30 01:55:28.787234876 (586h26m28.658317921s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787382778 (586h26m28.658465823s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787388960 (586h26m28.658471995s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787402736 (586h26m28.658485781s) [seg_pos={0 0}, upload=514094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:55:28.787461819 (586h26m28.658544864s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:55:28.787467429 (586h26m28.658550464s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:55:29.079428950 (586h26m28.950511995s) [upload=514094]: Done uploading segments... Dec 30 01:55:29.079451724 (586h26m28.950534769s) [upload=514094]: Upload finished. err: <nil> Duration: 13.068600194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 18.443838016s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 18.443858414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.514090991s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:56:15 UTC) orbiter warning: context deadline exceeded Duration: 3.514115978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...............XX.....X.X......X.X..XX.X....X..XXX..XX.....X] duration: 30.002570112s trace: https://jaeger.dev.storj.io/trace/1aa3b31e9fcd916c FAIL (2025-12-30 01:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ) context deadline exceeded ======LOGS====== 01:55:26.264693797 (586h26m26.135776842s) [upload=514094]: Starting upload Dec 30 01:55:26.264768260 (586h26m26.135851295s) [upload=514094]: Done starting upload Dec 30 01:55:26.264868770 (586h26m26.135951826s) [upload=514094]: Uploading segments... Dec 30 01:55:26.265054515 (586h26m26.13613755s) [upload=514094]: Got next segment. Inline: false Dec 30 01:55:26.265090924 (586h26m26.136173959s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:55:26.265096034 (586h26m26.136179069s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:55:26.508102931 (586h26m26.379185966s) [seg_pos={0 0}, upload=514094]: Begin upload segment... Dec 30 01:55:26.508145091 (586h26m26.379228126s) [seg_pos={0 0}, upload=514094]: Done begin upload segment. Dec 30 01:55:26.508171071 (586h26m26.379254106s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508250702 (586h26m26.379333767s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508748820 (586h26m26.379831855s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509099648 (586h26m26.380182683s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509393218 (586h26m26.380476253s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509679443 (586h26m26.380762478s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510089403 (586h26m26.381172448s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510402118 (586h26m26.381485163s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510723771 (586h26m26.381806806s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.511615949 (586h26m26.382699005s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.513000306 (586h26m26.384083351s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.514708609 (586h26m26.385791654s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.516561266 (586h26m26.387644301s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.518322920 (586h26m26.389405955s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.519685224 (586h26m26.390768269s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.520165879 (586h26m26.391248914s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:28.113351882 (586h26m27.984434917s) [upload=514094]: Next returned nil segment Dec 30 01:55:28.113366409 (586h26m27.984449444s) [upload=514094]: Waiting for error group managing segments... Dec 30 01:55:28.489714050 (586h26m28.360797085s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.495852478 (586h26m28.366935513s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.496659264 (586h26m28.367742299s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.499030169 (586h26m28.370113204s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.500746907 (586h26m28.371829982s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.504146861 (586h26m28.375229906s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512015273 (586h26m28.383098309s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512474377 (586h26m28.383557413s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.516709301 (586h26m28.387792346s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.519143556 (586h26m28.390226611s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.522158937 (586h26m28.393241972s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.717343337 (586h26m28.588426402s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786975082 (586h26m28.658058117s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786995331 (586h26m28.658078376s) [seg_pos={0 0}, upload=514094]: Segment reached optimal threshold of 13 pieces. Dec 30 01:55:28.787182006 (586h26m28.658265051s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787188408 (586h26m28.658271453s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787225378 (586h26m28.658308423s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection Dec 30 01:55:28.787234876 (586h26m28.658317921s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787382778 (586h26m28.658465823s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787388960 (586h26m28.658471995s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787402736 (586h26m28.658485781s) [seg_pos={0 0}, upload=514094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:55:28.787461819 (586h26m28.658544864s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:55:28.787467429 (586h26m28.658550464s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:55:29.079428950 (586h26m28.950511995s) [upload=514094]: Done uploading segments... Dec 30 01:55:29.079451724 (586h26m28.950534769s) [upload=514094]: Upload finished. err: <nil> Duration: 30.002590641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000995467s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:55:47 UTC) orbiter warning: context deadline exceeded Duration: 5.00101759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 27.253976648s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:55:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 27.254007576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................XX.....X.X......X.X..XX.X....X..XXX..XX.....] duration: 5.999536087s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:54:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 5.99956347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 34.578599096s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:54:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 34.578628822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 15.116016245s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:53:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.116047674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................XX.....X.X......X.X..XX.X....X..XXX..XX....] duration: 5.836291113s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:53:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 5.836316241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.................XX.....X.X......X.X..XX.X....X..XXX..XX...] duration: 5.908668123s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:53:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 5.908698811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 15.111184349s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:53:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.111214838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 37.096347917s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:52:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 37.096427979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.................XX.....X.X......X.X..XX.X....X..XXX..XX..] duration: 5.731328004s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:51:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 5.731353051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 38.107397566s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:51:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 38.107418556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.................XX.....X.X......X.X..XX.X....X..XXX..XX.] duration: 28.625392708s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 28.625409579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.................XX.....X.X......X.X..XX.X....X..XXX..XX] duration: 30.002934397s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 30.002944267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 40.548373056s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:51:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 40.548405297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X.................XX.....X.X......X.X..XX.X....X..XXX..X] duration: 30.003105691s trace: https://jaeger.dev.storj.io/trace/7bffd320238fb7e1 FAIL (2025-12-30 01:50:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: BGTD35LLIGCKNGEMV3HK6U5HVBMHTIPQQSSRY3YNXHMJAC76TLAQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: F6NAGG6TXKZRIPVWZRGYDHN6UMLE2GIPFDABRMY5U47FSXNZTVUA) context deadline exceeded ======LOGS====== Dec 30 01:50:10.004527167 (586h21m9.875610202s) [upload=514014]: Starting upload Dec 30 01:50:10.004625684 (586h21m9.875708729s) [upload=514014]: Done starting upload Dec 30 01:50:10.004772203 (586h21m9.875855248s) [upload=514014]: Uploading segments... Dec 30 01:50:10.004959129 (586h21m9.876042174s) [upload=514014]: Got next segment. Inline: false Dec 30 01:50:10.005025044 (586h21m9.876108079s) [upload=514014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:10.005035845 (586h21m9.87611888s) [upload=514014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:10.271658418 (586h21m10.142741463s) [seg_pos={0 0}, upload=514014]: Begin upload segment... Dec 30 01:50:10.271704556 (586h21m10.142787601s) [seg_pos={0 0}, upload=514014]: Done begin upload segment. Dec 30 01:50:10.271722079 (586h21m10.142805114s) [noise=true , address=145.79.227.15:4875 , piece_id=UBITTZW5EEP3PPNCO7T4NZNL6LZAUT5F252ZPTZHN3JOGSS3RS5Q, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.271745544 (586h21m10.142828579s) [noise=true , address=145.79.227.51:4409 , piece_id=H2FN6JBTDKT3QQD6UT4ZXORMUXBLMJHUVCJTWJIDGZ55EONCY4FA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.272160655 (586h21m10.1432437s) [noise=true , address=145.79.227.6:4258 , piece_id=D63YND2B3C245SIBYXE42L73R37IQFUO7IOOIQAYNK3A5LS4L4XA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.272933226 (586h21m10.144016261s) [noise=true , address=145.79.227.22:4509 , piece_id=WI5CFBUXQLRK4K6NSWOPTEH36FGKW3N6E4GVOX2DPHFIIMDPYDLQ, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.273324641 (586h21m10.144407676s) [noise=true , address=77.90.31.13:4626 , piece_id=BGTD35LLIGCKNGEMV3HK6U5HVBMHTIPQQSSRY3YNXHMJAC76TLAQ, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.273702360 (586h21m10.144785405s) [noise=true , address=145.79.227.38:4977 , piece_id=GSN2ROLRDOGMCHYFIG2JI6PGEAAGXZQXPSMQENLEWME3UZCIALKA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.274395099 (586h21m10.145478134s) [noise=true , address=45.91.116.5:4298 , piece_id=RA6N6DNTQ2Y63ZWWNSCUA5QXZOJI6M4LVLLUZX3WKS3I57ZJ3NRQ, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.274416860 (586h21m10.145499895s) [noise=true , address=77.90.31.12:4590 , piece_id=2D7FIQ4NVWFVUGFQAIKMTWULJ64VPAI5RZTJWJLAVTQZJGNJAICA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.274802004 (586h21m10.145885049s) [noise=true , address=77.90.31.14:4664 , piece_id=47XMJIOUEEMN7WHD34UUH7EKVFJIYR5OIR6M4ACDANV5OP7RYBKA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.274896774 (586h21m10.145979819s) [noise=true , address=145.79.227.33:4871 , piece_id=2YYAYLQJQ7JO6LJUKBUNP5M4UY2AINU7AW6ZFUJLBZY57ZBRKSHA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.275191245 (586h21m10.14627428s) [noise=true , address=45.91.116.10:4081 , piece_id=F6NAGG6TXKZRIPVWZRGYDHN6UMLE2GIPFDABRMY5U47FSXNZTVUA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.275455909 (586h21m10.146538954s) [noise=true , address=145.79.227.13:4802 , piece_id=AMP3I2YAILG6W6EZVQ3I5Q3D3WW46ARA3VVY55IFSS5BHYV2DONA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.275959648 (586h21m10.147042693s) [noise=true , address=145.79.227.47:4265 , piece_id=RY6NENXQWLWTGCD57K2SCGISGPVKAJRYDZSRENESPJNU42E7ASGQ, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.276678777 (586h21m10.147761822s) [noise=true , address=145.79.227.42:4123 , piece_id=PBNL447WDBCE7N7LAKHXWM4SZIGEXI7G436ZDLNXFKJ237TDSTIA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.276870712 (586h21m10.147953747s) [noise=true , address=45.91.116.13:4155 , piece_id=K2QZXEAIFYNDXFMPPVF4IEECTFYZMJIIRQO2JLOJPS3CDBBBGBMA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:10.277360735 (586h21m10.14844377s) [noise=true , address=145.79.227.14:4839 , piece_id=EBXZDV23MEG4B5Q5XOYVHQD6GQBHDV7AVYCDNPJ63T25WBQXAXIA, seg_pos={0 0}, upload=514014]: Uploading piece... Dec 30 01:50:11.866946022 (586h21m11.738029057s) [upload=514014]: Next returned nil segment Dec 30 01:50:11.866958737 (586h21m11.738041782s) [upload=514014]: Waiting for error group managing segments... Dec 30 01:50:12.246197091 (586h21m12.117280136s) [noise=true , address=145.79.227.15:4875 , piece_id=UBITTZW5EEP3PPNCO7T4NZNL6LZAUT5F252ZPTZHN3JOGSS3RS5Q, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.246627601 (586h21m12.117710636s) [noise=true , address=145.79.227.14:4839 , piece_id=EBXZDV23MEG4B5Q5XOYVHQD6GQBHDV7AVYCDNPJ63T25WBQXAXIA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.252767982 (586h21m12.123851027s) [noise=true , address=145.79.227.42:4123 , piece_id=PBNL447WDBCE7N7LAKHXWM4SZIGEXI7G436ZDLNXFKJ237TDSTIA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.254664193 (586h21m12.125747238s) [noise=true , address=145.79.227.6:4258 , piece_id=D63YND2B3C245SIBYXE42L73R37IQFUO7IOOIQAYNK3A5LS4L4XA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.263163366 (586h21m12.134246411s) [noise=true , address=145.79.227.51:4409 , piece_id=H2FN6JBTDKT3QQD6UT4ZXORMUXBLMJHUVCJTWJIDGZ55EONCY4FA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.264157700 (586h21m12.135240735s) [noise=true , address=145.79.227.38:4977 , piece_id=GSN2ROLRDOGMCHYFIG2JI6PGEAAGXZQXPSMQENLEWME3UZCIALKA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.265438177 (586h21m12.136521212s) [noise=true , address=145.79.227.47:4265 , piece_id=RY6NENXQWLWTGCD57K2SCGISGPVKAJRYDZSRENESPJNU42E7ASGQ, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.281121419 (586h21m12.152204464s) [noise=true , address=145.79.227.13:4802 , piece_id=AMP3I2YAILG6W6EZVQ3I5Q3D3WW46ARA3VVY55IFSS5BHYV2DONA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.285779058 (586h21m12.156862093s) [noise=true , address=145.79.227.22:4509 , piece_id=WI5CFBUXQLRK4K6NSWOPTEH36FGKW3N6E4GVOX2DPHFIIMDPYDLQ, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.439603694 (586h21m12.310686739s) [noise=true , address=77.90.31.13:4626 , piece_id=BGTD35LLIGCKNGEMV3HK6U5HVBMHTIPQQSSRY3YNXHMJAC76TLAQ, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.495093512 (586h21m12.366176557s) [noise=true , address=45.91.116.10:4081 , piece_id=F6NAGG6TXKZRIPVWZRGYDHN6UMLE2GIPFDABRMY5U47FSXNZTVUA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.508530007 (586h21m12.379613042s) [noise=true , address=45.91.116.5:4298 , piece_id=RA6N6DNTQ2Y63ZWWNSCUA5QXZOJI6M4LVLLUZX3WKS3I57ZJ3NRQ, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.518508427 (586h21m12.389591462s) [noise=true , address=45.91.116.13:4155 , piece_id=K2QZXEAIFYNDXFMPPVF4IEECTFYZMJIIRQO2JLOJPS3CDBBBGBMA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: <nil> Dec 30 01:50:12.518529937 (586h21m12.389612972s) [seg_pos={0 0}, upload=514014]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:12.518750668 (586h21m12.389833713s) [noise=true , address=145.79.227.33:4871 , piece_id=2YYAYLQJQ7JO6LJUKBUNP5M4UY2AINU7AW6ZFUJLBZY57ZBRKSHA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:12.518765376 (586h21m12.389848421s) [noise=true , address=145.79.227.33:4871 , piece_id=2YYAYLQJQ7JO6LJUKBUNP5M4UY2AINU7AW6ZFUJLBZY57ZBRKSHA, seg_pos={0 0}, upload=514014]: Long tail cancellation due to optimal threshold Dec 30 01:50:12.519064526 (586h21m12.390147561s) [noise=true , address=77.90.31.14:4664 , piece_id=47XMJIOUEEMN7WHD34UUH7EKVFJIYR5OIR6M4ACDANV5OP7RYBKA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:12.519071739 (586h21m12.390154774s) [noise=true , address=77.90.31.14:4664 , piece_id=47XMJIOUEEMN7WHD34UUH7EKVFJIYR5OIR6M4ACDANV5OP7RYBKA, seg_pos={0 0}, upload=514014]: Long tail cancellation due to optimal threshold Dec 30 01:50:12.519193350 (586h21m12.390276395s) [noise=true , address=77.90.31.12:4590 , piece_id=2D7FIQ4NVWFVUGFQAIKMTWULJ64VPAI5RZTJWJLAVTQZJGNJAICA, seg_pos={0 0}, upload=514014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:12.519199382 (586h21m12.390282427s) [noise=true , address=77.90.31.12:4590 , piece_id=2D7FIQ4NVWFVUGFQAIKMTWULJ64VPAI5RZTJWJLAVTQZJGNJAICA, seg_pos={0 0}, upload=514014]: Long tail cancellation due to optimal threshold Dec 30 01:50:12.519217978 (586h21m12.390301023s) [seg_pos={0 0}, upload=514014]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:12.519287380 (586h21m12.390370425s) [upload=514014]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:12.519292159 (586h21m12.390375204s) [upload=514014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:12.828810751 (586h21m12.699893786s) [upload=514014]: Done uploading segments... Dec 30 01:50:12.828847271 (586h21m12.699930316s) [upload=514014]: Upload finished. err: <nil> Duration: 30.003128324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 18.146303621s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 18.146333477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 57.895931752s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:49:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 57.895949365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X.................XX.....X.X......X.X..XX.X....X..XXX..] duration: 5.598705992s trace: https://jaeger.dev.storj.io/trace/1c8433dd90bbb4a0 FAIL (2025-12-30 01:49:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA) context deadline exceeded ======LOGS====== Dec 30 01:47:48.313965851 (586h18m48.185048886s) [upload=513979]: Starting upload Dec 30 01:47:48.314024222 (586h18m48.185107267s) [upload=513979]: Done starting upload Dec 30 01:47:48.314093083 (586h18m48.185176118s) [upload=513979]: Uploading segments... Dec 30 01:47:48.314221768 (586h18m48.185304803s) [upload=513979]: Got next segment. Inline: false Dec 30 01:47:48.314251405 (586h18m48.18533444s) [upload=513979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:47:48.314256795 (586h18m48.18533983s) [upload=513979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:47:48.561122590 (586h18m48.432205626s) [seg_pos={0 0}, upload=513979]: Begin upload segment... Dec 30 01:47:48.561185470 (586h18m48.432268505s) [seg_pos={0 0}, upload=513979]: Done begin upload segment. Dec 30 01:47:48.561243009 (586h18m48.432326054s) [noise=true , address=45.91.116.7:4011 , piece_id=VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.561247668 (586h18m48.432330703s) [noise=true , address=145.79.227.22:4511 , piece_id=FEXBOC352RDE6VT7FII5N7R3CRU7QRPJ6OJPQ3YEKVOIDIHGLTGQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562009890 (586h18m48.433092935s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562274694 (586h18m48.433357729s) [noise=true , address=145.79.227.38:4979 , piece_id=5VBLJ25I4J53OD3SX5VP566K5QYWSNSXDXSDTHXQBLHRM7TQQ7FQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562724611 (586h18m48.433807656s) [noise=true , address=77.90.31.13:4626 , piece_id=SYIRYTTX2RSTDW2Q3NSIHB4A6D2CSIDIDWD3MLCYLTAU2O56KLWQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562804713 (586h18m48.433887758s) [noise=true , address=145.79.227.45:4231 , piece_id=TRR6MXCJPA62W24RDN7U6NT3PT7LXEG4E3PQ6HJOCTYNDU5TSPXQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563167954 (586h18m48.43425099s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563562246 (586h18m48.434645281s) [noise=true , address=145.79.227.10:4402 , piece_id=32S2LFURC226DPYNAA5PIEMSUE2LHKA5LFFLWXN22SVAUAQUSMRA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563862076 (586h18m48.434945121s) [noise=true , address=77.90.31.16:4735 , piece_id=KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564204949 (586h18m48.435287984s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564491324 (586h18m48.435574369s) [noise=true , address=145.79.227.14:4838 , piece_id=JAXQ63EGHEQ3KHJPIPVN53GWJBZC3APIHWWFNW7YGAQ4ZAF7MM6A, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564799902 (586h18m48.435882937s) [noise=true , address=145.79.227.52:4447 , piece_id=IOJINU47REYNVBFHXJAHV2NN42Y5CYNLSLP32YNHG46DQ5DHJUHQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.565160148 (586h18m48.436243183s) [noise=true , address=145.79.227.51:4409 , piece_id=TENMTJI5FMLRYGO5ZU537VHSI77P4UDU3MQ4TLTF32PQDXZOS22A, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.565569037 (586h18m48.436652072s) [noise=true , address=145.79.227.47:4266 , piece_id=ECBQRMDNALN53AFE6UPAZD77XYETR6ZVY6B6ACRV2MU2E25JXSGA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.566517342 (586h18m48.437600377s) [noise=true , address=145.79.227.8:4329 , piece_id=C2WA6OFEYPQZGVM6UOOKTVQMB4KHXUQXNMCV5KKBS4HPCCOLZRBA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.567253644 (586h18m48.438336689s) [noise=true , address=145.79.227.48:4302 , piece_id=HQCGD6MVMO5RYQ3TMQWDJSJRPOPK6QY55RUKKIKMBCXRHMP6XYNQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:50.259213029 (586h18m50.130296075s) [upload=513979]: Next returned nil segment Dec 30 01:47:50.259234972 (586h18m50.130318017s) [upload=513979]: Waiting for error group managing segments... Dec 30 01:47:50.540048075 (586h18m50.41113111s) [noise=true , address=145.79.227.45:4231 , piece_id=TRR6MXCJPA62W24RDN7U6NT3PT7LXEG4E3PQ6HJOCTYNDU5TSPXQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.551086954 (586h18m50.422169989s) [noise=true , address=145.79.227.14:4838 , piece_id=JAXQ63EGHEQ3KHJPIPVN53GWJBZC3APIHWWFNW7YGAQ4ZAF7MM6A, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.553335776 (586h18m50.424418811s) [noise=true , address=145.79.227.8:4329 , piece_id=C2WA6OFEYPQZGVM6UOOKTVQMB4KHXUQXNMCV5KKBS4HPCCOLZRBA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.556424948 (586h18m50.427507993s) [noise=true , address=145.79.227.47:4266 , piece_id=ECBQRMDNALN53AFE6UPAZD77XYETR6ZVY6B6ACRV2MU2E25JXSGA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.561579764 (586h18m50.432662799s) [noise=true , address=145.79.227.52:4447 , piece_id=IOJINU47REYNVBFHXJAHV2NN42Y5CYNLSLP32YNHG46DQ5DHJUHQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.571018806 (586h18m50.442101841s) [noise=true , address=145.79.227.38:4979 , piece_id=5VBLJ25I4J53OD3SX5VP566K5QYWSNSXDXSDTHXQBLHRM7TQQ7FQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.571106644 (586h18m50.442189679s) [noise=true , address=145.79.227.10:4402 , piece_id=32S2LFURC226DPYNAA5PIEMSUE2LHKA5LFFLWXN22SVAUAQUSMRA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.578889743 (586h18m50.449972788s) [noise=true , address=145.79.227.51:4409 , piece_id=TENMTJI5FMLRYGO5ZU537VHSI77P4UDU3MQ4TLTF32PQDXZOS22A, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.584622409 (586h18m50.455705444s) [noise=true , address=145.79.227.22:4511 , piece_id=FEXBOC352RDE6VT7FII5N7R3CRU7QRPJ6OJPQ3YEKVOIDIHGLTGQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.584751805 (586h18m50.45583484s) [noise=true , address=145.79.227.48:4302 , piece_id=HQCGD6MVMO5RYQ3TMQWDJSJRPOPK6QY55RUKKIKMBCXRHMP6XYNQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.840364485 (586h18m50.71144752s) [noise=true , address=77.90.31.13:4626 , piece_id=SYIRYTTX2RSTDW2Q3NSIHB4A6D2CSIDIDWD3MLCYLTAU2O56KLWQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.849421400 (586h18m50.720504435s) [noise=true , address=77.90.31.16:4735 , piece_id=KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.919690910 (586h18m50.790773945s) [noise=true , address=45.91.116.7:4011 , piece_id=VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.919709325 (586h18m50.79079237s) [seg_pos={0 0}, upload=513979]: Segment reached optimal threshold of 13 pieces. Dec 30 01:47:50.919920988 (586h18m50.791004043s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:47:50.919932901 (586h18m50.791015956s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920063970 (586h18m50.791147005s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:55434->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 01:47:50.920072276 (586h18m50.791155311s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920137600 (586h18m50.791220635s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 01:47:50.920144744 (586h18m50.791227779s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920161155 (586h18m50.7912442s) [seg_pos={0 0}, upload=513979]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:47:50.920229625 (586h18m50.79131267s) [upload=513979]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:47:50.920233853 (586h18m50.791316888s) [upload=513979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:47:51.204790566 (586h18m51.075873611s) [upload=513979]: Done uploading segments... Dec 30 01:47:51.204815944 (586h18m51.075898979s) [upload=513979]: Upload finished. err: <nil> Duration: 5.598732802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.....X.................XX.....X.X......X.X..XX.X....X..XXX.] duration: 5.94193236s trace: https://jaeger.dev.storj.io/trace/1c8433dd90bbb4a0 FAIL (2025-12-30 01:48:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA) context deadline exceeded ======LOGS====== Dec 30 01:47:48.313965851 (586h18m48.185048886s) [upload=513979]: Starting upload Dec 30 01:47:48.314024222 (586h18m48.185107267s) [upload=513979]: Done starting upload Dec 30 01:47:48.314093083 (586h18m48.185176118s) [upload=513979]: Uploading segments... Dec 30 01:47:48.314221768 (586h18m48.185304803s) [upload=513979]: Got next segment. Inline: false Dec 30 01:47:48.314251405 (586h18m48.18533444s) [upload=513979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:47:48.314256795 (586h18m48.18533983s) [upload=513979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:47:48.561122590 (586h18m48.432205626s) [seg_pos={0 0}, upload=513979]: Begin upload segment... Dec 30 01:47:48.561185470 (586h18m48.432268505s) [seg_pos={0 0}, upload=513979]: Done begin upload segment. Dec 30 01:47:48.561243009 (586h18m48.432326054s) [noise=true , address=45.91.116.7:4011 , piece_id=VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.561247668 (586h18m48.432330703s) [noise=true , address=145.79.227.22:4511 , piece_id=FEXBOC352RDE6VT7FII5N7R3CRU7QRPJ6OJPQ3YEKVOIDIHGLTGQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562009890 (586h18m48.433092935s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562274694 (586h18m48.433357729s) [noise=true , address=145.79.227.38:4979 , piece_id=5VBLJ25I4J53OD3SX5VP566K5QYWSNSXDXSDTHXQBLHRM7TQQ7FQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562724611 (586h18m48.433807656s) [noise=true , address=77.90.31.13:4626 , piece_id=SYIRYTTX2RSTDW2Q3NSIHB4A6D2CSIDIDWD3MLCYLTAU2O56KLWQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562804713 (586h18m48.433887758s) [noise=true , address=145.79.227.45:4231 , piece_id=TRR6MXCJPA62W24RDN7U6NT3PT7LXEG4E3PQ6HJOCTYNDU5TSPXQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563167954 (586h18m48.43425099s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563562246 (586h18m48.434645281s) [noise=true , address=145.79.227.10:4402 , piece_id=32S2LFURC226DPYNAA5PIEMSUE2LHKA5LFFLWXN22SVAUAQUSMRA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563862076 (586h18m48.434945121s) [noise=true , address=77.90.31.16:4735 , piece_id=KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564204949 (586h18m48.435287984s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564491324 (586h18m48.435574369s) [noise=true , address=145.79.227.14:4838 , piece_id=JAXQ63EGHEQ3KHJPIPVN53GWJBZC3APIHWWFNW7YGAQ4ZAF7MM6A, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564799902 (586h18m48.435882937s) [noise=true , address=145.79.227.52:4447 , piece_id=IOJINU47REYNVBFHXJAHV2NN42Y5CYNLSLP32YNHG46DQ5DHJUHQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.565160148 (586h18m48.436243183s) [noise=true , address=145.79.227.51:4409 , piece_id=TENMTJI5FMLRYGO5ZU537VHSI77P4UDU3MQ4TLTF32PQDXZOS22A, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.565569037 (586h18m48.436652072s) [noise=true , address=145.79.227.47:4266 , piece_id=ECBQRMDNALN53AFE6UPAZD77XYETR6ZVY6B6ACRV2MU2E25JXSGA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.566517342 (586h18m48.437600377s) [noise=true , address=145.79.227.8:4329 , piece_id=C2WA6OFEYPQZGVM6UOOKTVQMB4KHXUQXNMCV5KKBS4HPCCOLZRBA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.567253644 (586h18m48.438336689s) [noise=true , address=145.79.227.48:4302 , piece_id=HQCGD6MVMO5RYQ3TMQWDJSJRPOPK6QY55RUKKIKMBCXRHMP6XYNQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:50.259213029 (586h18m50.130296075s) [upload=513979]: Next returned nil segment Dec 30 01:47:50.259234972 (586h18m50.130318017s) [upload=513979]: Waiting for error group managing segments... Dec 30 01:47:50.540048075 (586h18m50.41113111s) [noise=true , address=145.79.227.45:4231 , piece_id=TRR6MXCJPA62W24RDN7U6NT3PT7LXEG4E3PQ6HJOCTYNDU5TSPXQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.551086954 (586h18m50.422169989s) [noise=true , address=145.79.227.14:4838 , piece_id=JAXQ63EGHEQ3KHJPIPVN53GWJBZC3APIHWWFNW7YGAQ4ZAF7MM6A, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.553335776 (586h18m50.424418811s) [noise=true , address=145.79.227.8:4329 , piece_id=C2WA6OFEYPQZGVM6UOOKTVQMB4KHXUQXNMCV5KKBS4HPCCOLZRBA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.556424948 (586h18m50.427507993s) [noise=true , address=145.79.227.47:4266 , piece_id=ECBQRMDNALN53AFE6UPAZD77XYETR6ZVY6B6ACRV2MU2E25JXSGA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.561579764 (586h18m50.432662799s) [noise=true , address=145.79.227.52:4447 , piece_id=IOJINU47REYNVBFHXJAHV2NN42Y5CYNLSLP32YNHG46DQ5DHJUHQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.571018806 (586h18m50.442101841s) [noise=true , address=145.79.227.38:4979 , piece_id=5VBLJ25I4J53OD3SX5VP566K5QYWSNSXDXSDTHXQBLHRM7TQQ7FQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.571106644 (586h18m50.442189679s) [noise=true , address=145.79.227.10:4402 , piece_id=32S2LFURC226DPYNAA5PIEMSUE2LHKA5LFFLWXN22SVAUAQUSMRA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.578889743 (586h18m50.449972788s) [noise=true , address=145.79.227.51:4409 , piece_id=TENMTJI5FMLRYGO5ZU537VHSI77P4UDU3MQ4TLTF32PQDXZOS22A, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.584622409 (586h18m50.455705444s) [noise=true , address=145.79.227.22:4511 , piece_id=FEXBOC352RDE6VT7FII5N7R3CRU7QRPJ6OJPQ3YEKVOIDIHGLTGQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.584751805 (586h18m50.45583484s) [noise=true , address=145.79.227.48:4302 , piece_id=HQCGD6MVMO5RYQ3TMQWDJSJRPOPK6QY55RUKKIKMBCXRHMP6XYNQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.840364485 (586h18m50.71144752s) [noise=true , address=77.90.31.13:4626 , piece_id=SYIRYTTX2RSTDW2Q3NSIHB4A6D2CSIDIDWD3MLCYLTAU2O56KLWQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.849421400 (586h18m50.720504435s) [noise=true , address=77.90.31.16:4735 , piece_id=KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.919690910 (586h18m50.790773945s) [noise=true , address=45.91.116.7:4011 , piece_id=VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.919709325 (586h18m50.79079237s) [seg_pos={0 0}, upload=513979]: Segment reached optimal threshold of 13 pieces. Dec 30 01:47:50.919920988 (586h18m50.791004043s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:47:50.919932901 (586h18m50.791015956s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920063970 (586h18m50.791147005s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:55434->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 01:47:50.920072276 (586h18m50.791155311s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920137600 (586h18m50.791220635s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 01:47:50.920144744 (586h18m50.791227779s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920161155 (586h18m50.7912442s) [seg_pos={0 0}, upload=513979]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:47:50.920229625 (586h18m50.79131267s) [upload=513979]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:47:50.920233853 (586h18m50.791316888s) [upload=513979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:47:51.204790566 (586h18m51.075873611s) [upload=513979]: Done uploading segments... Dec 30 01:47:51.204815944 (586h18m51.075898979s) [upload=513979]: Upload finished. err: <nil> Duration: 5.94194782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 47.66128568s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:48:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 47.661305918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.....X.................XX.....X.X......X.X..XX.X....X..XXX] duration: 30.002383434s trace: https://jaeger.dev.storj.io/trace/1c8433dd90bbb4a0 FAIL (2025-12-30 01:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA) context deadline exceeded ======LOGS====== Dec 30 01:47:48.313965851 (586h18m48.185048886s) [upload=513979]: Starting upload Dec 30 01:47:48.314024222 (586h18m48.185107267s) [upload=513979]: Done starting upload Dec 30 01:47:48.314093083 (586h18m48.185176118s) [upload=513979]: Uploading segments... Dec 30 01:47:48.314221768 (586h18m48.185304803s) [upload=513979]: Got next segment. Inline: false Dec 30 01:47:48.314251405 (586h18m48.18533444s) [upload=513979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:47:48.314256795 (586h18m48.18533983s) [upload=513979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:47:48.561122590 (586h18m48.432205626s) [seg_pos={0 0}, upload=513979]: Begin upload segment... Dec 30 01:47:48.561185470 (586h18m48.432268505s) [seg_pos={0 0}, upload=513979]: Done begin upload segment. Dec 30 01:47:48.561243009 (586h18m48.432326054s) [noise=true , address=45.91.116.7:4011 , piece_id=VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.561247668 (586h18m48.432330703s) [noise=true , address=145.79.227.22:4511 , piece_id=FEXBOC352RDE6VT7FII5N7R3CRU7QRPJ6OJPQ3YEKVOIDIHGLTGQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562009890 (586h18m48.433092935s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562274694 (586h18m48.433357729s) [noise=true , address=145.79.227.38:4979 , piece_id=5VBLJ25I4J53OD3SX5VP566K5QYWSNSXDXSDTHXQBLHRM7TQQ7FQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562724611 (586h18m48.433807656s) [noise=true , address=77.90.31.13:4626 , piece_id=SYIRYTTX2RSTDW2Q3NSIHB4A6D2CSIDIDWD3MLCYLTAU2O56KLWQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.562804713 (586h18m48.433887758s) [noise=true , address=145.79.227.45:4231 , piece_id=TRR6MXCJPA62W24RDN7U6NT3PT7LXEG4E3PQ6HJOCTYNDU5TSPXQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563167954 (586h18m48.43425099s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563562246 (586h18m48.434645281s) [noise=true , address=145.79.227.10:4402 , piece_id=32S2LFURC226DPYNAA5PIEMSUE2LHKA5LFFLWXN22SVAUAQUSMRA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.563862076 (586h18m48.434945121s) [noise=true , address=77.90.31.16:4735 , piece_id=KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564204949 (586h18m48.435287984s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564491324 (586h18m48.435574369s) [noise=true , address=145.79.227.14:4838 , piece_id=JAXQ63EGHEQ3KHJPIPVN53GWJBZC3APIHWWFNW7YGAQ4ZAF7MM6A, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.564799902 (586h18m48.435882937s) [noise=true , address=145.79.227.52:4447 , piece_id=IOJINU47REYNVBFHXJAHV2NN42Y5CYNLSLP32YNHG46DQ5DHJUHQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.565160148 (586h18m48.436243183s) [noise=true , address=145.79.227.51:4409 , piece_id=TENMTJI5FMLRYGO5ZU537VHSI77P4UDU3MQ4TLTF32PQDXZOS22A, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.565569037 (586h18m48.436652072s) [noise=true , address=145.79.227.47:4266 , piece_id=ECBQRMDNALN53AFE6UPAZD77XYETR6ZVY6B6ACRV2MU2E25JXSGA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.566517342 (586h18m48.437600377s) [noise=true , address=145.79.227.8:4329 , piece_id=C2WA6OFEYPQZGVM6UOOKTVQMB4KHXUQXNMCV5KKBS4HPCCOLZRBA, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:48.567253644 (586h18m48.438336689s) [noise=true , address=145.79.227.48:4302 , piece_id=HQCGD6MVMO5RYQ3TMQWDJSJRPOPK6QY55RUKKIKMBCXRHMP6XYNQ, seg_pos={0 0}, upload=513979]: Uploading piece... Dec 30 01:47:50.259213029 (586h18m50.130296075s) [upload=513979]: Next returned nil segment Dec 30 01:47:50.259234972 (586h18m50.130318017s) [upload=513979]: Waiting for error group managing segments... Dec 30 01:47:50.540048075 (586h18m50.41113111s) [noise=true , address=145.79.227.45:4231 , piece_id=TRR6MXCJPA62W24RDN7U6NT3PT7LXEG4E3PQ6HJOCTYNDU5TSPXQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.551086954 (586h18m50.422169989s) [noise=true , address=145.79.227.14:4838 , piece_id=JAXQ63EGHEQ3KHJPIPVN53GWJBZC3APIHWWFNW7YGAQ4ZAF7MM6A, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.553335776 (586h18m50.424418811s) [noise=true , address=145.79.227.8:4329 , piece_id=C2WA6OFEYPQZGVM6UOOKTVQMB4KHXUQXNMCV5KKBS4HPCCOLZRBA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.556424948 (586h18m50.427507993s) [noise=true , address=145.79.227.47:4266 , piece_id=ECBQRMDNALN53AFE6UPAZD77XYETR6ZVY6B6ACRV2MU2E25JXSGA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.561579764 (586h18m50.432662799s) [noise=true , address=145.79.227.52:4447 , piece_id=IOJINU47REYNVBFHXJAHV2NN42Y5CYNLSLP32YNHG46DQ5DHJUHQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.571018806 (586h18m50.442101841s) [noise=true , address=145.79.227.38:4979 , piece_id=5VBLJ25I4J53OD3SX5VP566K5QYWSNSXDXSDTHXQBLHRM7TQQ7FQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.571106644 (586h18m50.442189679s) [noise=true , address=145.79.227.10:4402 , piece_id=32S2LFURC226DPYNAA5PIEMSUE2LHKA5LFFLWXN22SVAUAQUSMRA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.578889743 (586h18m50.449972788s) [noise=true , address=145.79.227.51:4409 , piece_id=TENMTJI5FMLRYGO5ZU537VHSI77P4UDU3MQ4TLTF32PQDXZOS22A, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.584622409 (586h18m50.455705444s) [noise=true , address=145.79.227.22:4511 , piece_id=FEXBOC352RDE6VT7FII5N7R3CRU7QRPJ6OJPQ3YEKVOIDIHGLTGQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.584751805 (586h18m50.45583484s) [noise=true , address=145.79.227.48:4302 , piece_id=HQCGD6MVMO5RYQ3TMQWDJSJRPOPK6QY55RUKKIKMBCXRHMP6XYNQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.840364485 (586h18m50.71144752s) [noise=true , address=77.90.31.13:4626 , piece_id=SYIRYTTX2RSTDW2Q3NSIHB4A6D2CSIDIDWD3MLCYLTAU2O56KLWQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.849421400 (586h18m50.720504435s) [noise=true , address=77.90.31.16:4735 , piece_id=KESTUQGTNOZ2YOYC2ZPO6JJLWJZJ466WLQ2PZNGLA7RAWU4AV7WA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.919690910 (586h18m50.790773945s) [noise=true , address=45.91.116.7:4011 , piece_id=VTJKX5RGJAXHDSLX46G3GJEEIVFARLFQ6S5HKEW7EOB2DGDV4JOA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: <nil> Dec 30 01:47:50.919709325 (586h18m50.79079237s) [seg_pos={0 0}, upload=513979]: Segment reached optimal threshold of 13 pieces. Dec 30 01:47:50.919920988 (586h18m50.791004043s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:47:50.919932901 (586h18m50.791015956s) [noise=true , address=145.79.227.34:4907 , piece_id=RKIYYYZTYATOFN52KGAHVCSHVB56ZF6F2DJNKSTFUVM6D3PVNRDQ, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920063970 (586h18m50.791147005s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: read tcp 188.166.225.170:55434->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 01:47:50.920072276 (586h18m50.791155311s) [noise=true , address=77.90.31.11:4554 , piece_id=KQLWKMKHAM4JW62H2T7N6WGIJAV4PSI7AUA6ZMG4V5IAFXJQVEGQ, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920137600 (586h18m50.791220635s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 01:47:50.920144744 (586h18m50.791227779s) [noise=true , address=145.79.227.49:4338 , piece_id=4MWFMQKC4Z73R5SVRXLPVN2GVJYN7S7FKORUVGYZ2COCCFL3EHIA, seg_pos={0 0}, upload=513979]: Long tail cancellation due to optimal threshold Dec 30 01:47:50.920161155 (586h18m50.7912442s) [seg_pos={0 0}, upload=513979]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:47:50.920229625 (586h18m50.79131267s) [upload=513979]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:47:50.920233853 (586h18m50.791316888s) [upload=513979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:47:51.204790566 (586h18m51.075873611s) [upload=513979]: Done uploading segments... Dec 30 01:47:51.204815944 (586h18m51.075898979s) [upload=513979]: Upload finished. err: <nil> Duration: 30.002404334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 26.830931842s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:47:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 26.830960708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X.....X.................XX.....X.X......X.X..XX.X....X..XX] duration: 30.005666057s trace: https://jaeger.dev.storj.io/trace/45686df987c9659a FAIL (2025-12-30 01:47:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: Z6FVQ3ZGLJDK5YBGAKRYM27YWEOATEBM3RXC7UB5BSU2RGEN74TA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: JSAOE7PUAYZECNGTLCHO53A4IRISJXLHGMFOMKMFMNMSARP5ESOQ) context deadline exceeded ======LOGS====== Dec 30 01:46:52.565973652 (586h17m52.437056687s) [upload=513967]: Starting upload Dec 30 01:46:52.566076749 (586h17m52.437159784s) [upload=513967]: Done starting upload Dec 30 01:46:52.566171379 (586h17m52.437254424s) [upload=513967]: Uploading segments... Dec 30 01:46:52.566337926 (586h17m52.437420972s) [upload=513967]: Got next segment. Inline: false Dec 30 01:46:52.566391999 (586h17m52.437475044s) [upload=513967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:46:52.566398110 (586h17m52.437481155s) [upload=513967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:46:52.824061887 (586h17m52.695144922s) [seg_pos={0 0}, upload=513967]: Begin upload segment... Dec 30 01:46:52.824106381 (586h17m52.695189426s) [seg_pos={0 0}, upload=513967]: Done begin upload segment. Dec 30 01:46:52.824134114 (586h17m52.695217149s) [noise=true , address=145.79.227.2:4115 , piece_id=GH56FPFYC43I564BF4GKFREXKWOGSJ5BV2TEHWSSVOLDNJBRUBOQ, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.824212363 (586h17m52.695295438s) [noise=true , address=145.79.227.52:4447 , piece_id=MPLVLLHXTVOS5DX5V23I7LOXW7DP24J6ZTHYYKXEGLIZW2QCE6CQ, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.824852432 (586h17m52.695935468s) [noise=true , address=77.90.31.17:4770 , piece_id=VWJLWDMSBGN4XSUOVTJCTJBC3XGNNGDDSJWD65VLFQVHURXLZVQA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.826130155 (586h17m52.69721322s) [noise=true , address=45.91.116.10:4082 , piece_id=5H56MCI7FCGZ7OM4767EDDWKAF3PNURR3MTPRQTWDS6KBTVZYR2A, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.827231793 (586h17m52.698314838s) [noise=true , address=145.79.227.10:4402 , piece_id=F733BHAGDZBV4MKOVWRNUHE3E5Y7P6S4J57NHFZVHDZCZ6ZZPGTA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.827262912 (586h17m52.698345947s) [noise=true , address=145.79.227.9:4367 , piece_id=QYVWNAOUH43YKIVYQOVNROGYZOYFYUAUHYCZJMX4TQ774KI7R2PQ, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.828702423 (586h17m52.699785458s) [noise=true , address=145.79.227.36:4518 , piece_id=PQMRRP4KZ4VFDYLXH5TJ6UXAIM2NCYR62BECLP6PMX36OCP2DAPA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.828985301 (586h17m52.700068347s) [noise=true , address=45.91.116.8:4048 , piece_id=MTVJAYX5DVXEYGLIWWZOX3IQVYD45XYXLNS6TLAJ5MVNU5L52BGQ, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.830066430 (586h17m52.701149465s) [noise=true , address=145.79.227.39:4015 , piece_id=RTZU3LVURIYNAWPG6HOKAQ7AVPTS6MDFF2NBCZOK6DTHU6Z7COBA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.830157062 (586h17m52.701240127s) [noise=true , address=145.79.227.17:4972 , piece_id=E64QBVN7K4TGVN27353LZ4D2LFXOGKHBQUG7KWQQ2INBBUOJ6QMQ, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.830574687 (586h17m52.701657722s) [noise=true , address=145.79.227.51:4411 , piece_id=65UNSLEP53KLXCT22EKNEQIQNITNKUQO56JULYGRAYI6LQUDZ6TA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.831112432 (586h17m52.702195477s) [noise=true , address=145.79.227.24:4582 , piece_id=LFZV2NLB6B3HYNZPYQ7OC2MH7BPZAUXBLZDFBBB2J2IXU62T6NZA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.831236688 (586h17m52.702319733s) [noise=true , address=45.91.116.12:4227 , piece_id=Z6FVQ3ZGLJDK5YBGAKRYM27YWEOATEBM3RXC7UB5BSU2RGEN74TA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.832455428 (586h17m52.703538464s) [noise=true , address=145.79.227.26:4654 , piece_id=J7FFXFTGSMZFRIFKGS7KTMUQCIYZIGITSB7NOJVCATXNSD5D44JA, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.832664847 (586h17m52.703747882s) [noise=true , address=45.91.116.5:4297 , piece_id=JSAOE7PUAYZECNGTLCHO53A4IRISJXLHGMFOMKMFMNMSARP5ESOQ, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:52.833297062 (586h17m52.704380097s) [noise=true , address=45.91.116.9:4264 , piece_id=O2Z66HBQRTH5MLT6Q66RQNJGHINW7R2CY5XA5PJVMZWJ6WEPIOBQ, seg_pos={0 0}, upload=513967]: Uploading piece... Dec 30 01:46:54.483677849 (586h17m54.354760884s) [upload=513967]: Next returned nil segment Dec 30 01:46:54.483691735 (586h17m54.35477477s) [upload=513967]: Waiting for error group managing segments... Dec 30 01:46:54.816027333 (586h17m54.687110368s) [noise=true , address=145.79.227.9:4367 , piece_id=QYVWNAOUH43YKIVYQOVNROGYZOYFYUAUHYCZJMX4TQ774KI7R2PQ, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.818002484 (586h17m54.689085519s) [noise=true , address=145.79.227.17:4972 , piece_id=E64QBVN7K4TGVN27353LZ4D2LFXOGKHBQUG7KWQQ2INBBUOJ6QMQ, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.822787184 (586h17m54.693870219s) [noise=true , address=145.79.227.52:4447 , piece_id=MPLVLLHXTVOS5DX5V23I7LOXW7DP24J6ZTHYYKXEGLIZW2QCE6CQ, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.825530337 (586h17m54.696613382s) [noise=true , address=145.79.227.10:4402 , piece_id=F733BHAGDZBV4MKOVWRNUHE3E5Y7P6S4J57NHFZVHDZCZ6ZZPGTA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.830419116 (586h17m54.701502151s) [noise=true , address=145.79.227.2:4115 , piece_id=GH56FPFYC43I564BF4GKFREXKWOGSJ5BV2TEHWSSVOLDNJBRUBOQ, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.841526615 (586h17m54.71260966s) [noise=true , address=145.79.227.39:4015 , piece_id=RTZU3LVURIYNAWPG6HOKAQ7AVPTS6MDFF2NBCZOK6DTHU6Z7COBA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.847272135 (586h17m54.718355181s) [noise=true , address=145.79.227.51:4411 , piece_id=65UNSLEP53KLXCT22EKNEQIQNITNKUQO56JULYGRAYI6LQUDZ6TA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.850978302 (586h17m54.722061347s) [noise=true , address=145.79.227.24:4582 , piece_id=LFZV2NLB6B3HYNZPYQ7OC2MH7BPZAUXBLZDFBBB2J2IXU62T6NZA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.851267594 (586h17m54.722350639s) [noise=true , address=145.79.227.26:4654 , piece_id=J7FFXFTGSMZFRIFKGS7KTMUQCIYZIGITSB7NOJVCATXNSD5D44JA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:54.852615159 (586h17m54.723698194s) [noise=true , address=145.79.227.36:4518 , piece_id=PQMRRP4KZ4VFDYLXH5TJ6UXAIM2NCYR62BECLP6PMX36OCP2DAPA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:55.089289870 (586h17m54.960372905s) [noise=true , address=77.90.31.17:4770 , piece_id=VWJLWDMSBGN4XSUOVTJCTJBC3XGNNGDDSJWD65VLFQVHURXLZVQA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:55.132773827 (586h17m55.003856873s) [noise=true , address=45.91.116.5:4297 , piece_id=JSAOE7PUAYZECNGTLCHO53A4IRISJXLHGMFOMKMFMNMSARP5ESOQ, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:55.143351608 (586h17m55.014434643s) [noise=true , address=45.91.116.12:4227 , piece_id=Z6FVQ3ZGLJDK5YBGAKRYM27YWEOATEBM3RXC7UB5BSU2RGEN74TA, seg_pos={0 0}, upload=513967]: Done uploading piece. err: <nil> Dec 30 01:46:55.143367478 (586h17m55.014450513s) [seg_pos={0 0}, upload=513967]: Segment reached optimal threshold of 13 pieces. Dec 30 01:46:55.143524336 (586h17m55.014607372s) [noise=true , address=45.91.116.9:4264 , piece_id=O2Z66HBQRTH5MLT6Q66RQNJGHINW7R2CY5XA5PJVMZWJ6WEPIOBQ, seg_pos={0 0}, upload=513967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 01:46:55.143531140 (586h17m55.014614175s) [noise=true , address=45.91.116.9:4264 , piece_id=O2Z66HBQRTH5MLT6Q66RQNJGHINW7R2CY5XA5PJVMZWJ6WEPIOBQ, seg_pos={0 0}, upload=513967]: Long tail cancellation due to optimal threshold Dec 30 01:46:55.143715822 (586h17m55.014798857s) [noise=true , address=45.91.116.8:4048 , piece_id=MTVJAYX5DVXEYGLIWWZOX3IQVYD45XYXLNS6TLAJ5MVNU5L52BGQ, seg_pos={0 0}, upload=513967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 01:46:55.143722935 (586h17m55.01480597s) [noise=true , address=45.91.116.8:4048 , piece_id=MTVJAYX5DVXEYGLIWWZOX3IQVYD45XYXLNS6TLAJ5MVNU5L52BGQ, seg_pos={0 0}, upload=513967]: Long tail cancellation due to optimal threshold Dec 30 01:46:55.143794591 (586h17m55.014877626s) [noise=true , address=45.91.116.10:4082 , piece_id=5H56MCI7FCGZ7OM4767EDDWKAF3PNURR3MTPRQTWDS6KBTVZYR2A, seg_pos={0 0}, upload=513967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 30 01:46:55.143814800 (586h17m55.014897835s) [noise=true , address=45.91.116.10:4082 , piece_id=5H56MCI7FCGZ7OM4767EDDWKAF3PNURR3MTPRQTWDS6KBTVZYR2A, seg_pos={0 0}, upload=513967]: Long tail cancellation due to optimal threshold Dec 30 01:46:55.143882700 (586h17m55.014965735s) [seg_pos={0 0}, upload=513967]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:46:55.143957111 (586h17m55.015040146s) [upload=513967]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:46:55.143963293 (586h17m55.015046328s) [upload=513967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:46:55.441572930 (586h17m55.312655965s) [upload=513967]: Done uploading segments... Dec 30 01:46:55.441596774 (586h17m55.312679809s) [upload=513967]: Upload finished. err: <nil> Duration: 30.006536644s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 41.733699437s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:46:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 41.733728481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X.....X.................XX.....X.X......X.X..XX.X....X..X] duration: 30.003033421s trace: https://jaeger.dev.storj.io/trace/723362cd9d39ab73 FAIL (2025-12-30 01:46:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PD42BVDY7LKYZ52I4YZUTKW43TFJLRIM75IFUUPAVTXCV2APFYQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: SEKS4JREG6VTX25AFMZO43SZXLTNIKP6JUR26OHZWXIM6PEVCMUA) context deadline exceeded ======LOGS====== Dec 30 01:46:05.863238879 (586h17m5.734321924s) [upload=513956]: Starting upload Dec 30 01:46:05.863337718 (586h17m5.734420763s) [upload=513956]: Done starting upload Dec 30 01:46:05.863611760 (586h17m5.734694805s) [upload=513956]: Uploading segments... Dec 30 01:46:05.863660923 (586h17m5.734743978s) [upload=513956]: Got next segment. Inline: false Dec 30 01:46:05.863695348 (586h17m5.734778403s) [upload=513956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:46:05.863700678 (586h17m5.734783733s) [upload=513956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:46:06.103406000 (586h17m5.974489035s) [seg_pos={0 0}, upload=513956]: Begin upload segment... Dec 30 01:46:06.103456346 (586h17m5.974539381s) [seg_pos={0 0}, upload=513956]: Done begin upload segment. Dec 30 01:46:06.103496773 (586h17m5.974579808s) [noise=true , address=145.79.227.3:4150 , piece_id=L6DXRG64C3FPDTVG3YZWMZFYKMOCZCCLIGKCGDBIEFOVBTTWMUSA, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.103528443 (586h17m5.974611478s) [noise=true , address=77.90.31.18:4805 , piece_id=VE7HSMAWGMAXPSRUYGSGYR27N4Q273A4UYIK5WSLTWGSDM57CHOQ, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.104051239 (586h17m5.975134274s) [noise=true , address=145.79.227.12:4765 , piece_id=TRM5VGWQQF7Z7PGJVWVRQM6CR3JMUOIDPDOX7V4CO67FTTPWURGQ, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.104128295 (586h17m5.97521134s) [noise=true , address=145.79.227.7:4294 , piece_id=RK6YRDP7N636V66YXGPVGQMXAT2DZZ2K7OPWEKPEJTQ2IM4TVITA, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.104420742 (586h17m5.975503787s) [noise=true , address=145.79.227.10:4402 , piece_id=NRS5RYXJWVNHZLB3CP72YKK5KA72OPJAALGHSPQ5Z72QVEM3U7SA, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.104489984 (586h17m5.975573019s) [noise=true , address=145.79.227.14:4838 , piece_id=SJDVGHJY3M55I7T7RLQDNEU3ATARC5HAR4P25PJ5C2WRL6CXO7EA, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.104799904 (586h17m5.975882939s) [noise=true , address=77.90.31.14:4662 , piece_id=PKH6TRZOMMDKQ4VRL4WJJYVSANEBUWIAC7DWMOIAKHGONJRICG6A, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.104857003 (586h17m5.975940048s) [noise=true , address=145.79.227.32:4834 , piece_id=VSRZAXXNUSKIN3TVENMTC5T6OXNYOHXAW4DX3ODA4DDBW3OWU5KQ, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.105076851 (586h17m5.976159886s) [noise=true , address=145.79.227.45:4229 , piece_id=VSCFYPE7OQL2RFLZEODFG64KVKKVDP5P27S3XEVHI3JQ3MVA3V3Q, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.105326256 (586h17m5.976409301s) [noise=true , address=45.91.116.14:4192 , piece_id=37V7RQYB7S6DFSW7PEOBU6BXX5EQ4IP7O53TT5WTECGJ4JJKUOCA, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.105372575 (586h17m5.97645561s) [noise=true , address=77.90.31.13:4629 , piece_id=OWCLLN5DLJ7HG7MIARH4Y74OHHU3DQ4KHHUZOT3LCCYYN4PZHDTQ, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.105717512 (586h17m5.976800547s) [noise=true , address=45.91.116.10:4083 , piece_id=5PD42BVDY7LKYZ52I4YZUTKW43TFJLRIM75IFUUPAVTXCV2APFYQ, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.106301403 (586h17m5.977384448s) [noise=true , address=145.79.227.39:4013 , piece_id=UA666QW4AWRPUFHTOXYRPOI6CVHJWXIXRV3WRWFSHIATV2GJTJTQ, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.106672219 (586h17m5.977755254s) [noise=true , address=45.91.116.11:4119 , piece_id=SEKS4JREG6VTX25AFMZO43SZXLTNIKP6JUR26OHZWXIM6PEVCMUA, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.106990776 (586h17m5.978073841s) [noise=true , address=145.79.227.15:4873 , piece_id=TJXL6WX4RLQJ7WEOGUFMKXP6ELVF2O3XEAWY6UCHMJEAAY5IOK4Q, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:06.109259135 (586h17m5.98034217s) [noise=true , address=145.79.227.16:4912 , piece_id=6MULM627IZG6LXYHWWZ7PQ5RNX4NJNYSP23RP54FGEO53I3D2S6Q, seg_pos={0 0}, upload=513956]: Uploading piece... Dec 30 01:46:07.711183367 (586h17m7.582266402s) [upload=513956]: Next returned nil segment Dec 30 01:46:07.711197374 (586h17m7.582280419s) [upload=513956]: Waiting for error group managing segments... Dec 30 01:46:08.092163899 (586h17m7.963246934s) [noise=true , address=145.79.227.15:4873 , piece_id=TJXL6WX4RLQJ7WEOGUFMKXP6ELVF2O3XEAWY6UCHMJEAAY5IOK4Q, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.092266895 (586h17m7.96334993s) [noise=true , address=145.79.227.14:4838 , piece_id=SJDVGHJY3M55I7T7RLQDNEU3ATARC5HAR4P25PJ5C2WRL6CXO7EA, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.093755450 (586h17m7.964838485s) [noise=true , address=145.79.227.16:4912 , piece_id=6MULM627IZG6LXYHWWZ7PQ5RNX4NJNYSP23RP54FGEO53I3D2S6Q, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.095896596 (586h17m7.966979631s) [noise=true , address=145.79.227.3:4150 , piece_id=L6DXRG64C3FPDTVG3YZWMZFYKMOCZCCLIGKCGDBIEFOVBTTWMUSA, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.097962570 (586h17m7.969045605s) [noise=true , address=145.79.227.32:4834 , piece_id=VSRZAXXNUSKIN3TVENMTC5T6OXNYOHXAW4DX3ODA4DDBW3OWU5KQ, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.100820682 (586h17m7.971903727s) [noise=true , address=145.79.227.12:4765 , piece_id=TRM5VGWQQF7Z7PGJVWVRQM6CR3JMUOIDPDOX7V4CO67FTTPWURGQ, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.118527858 (586h17m7.989610903s) [noise=true , address=145.79.227.7:4294 , piece_id=RK6YRDP7N636V66YXGPVGQMXAT2DZZ2K7OPWEKPEJTQ2IM4TVITA, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.119020055 (586h17m7.99010309s) [noise=true , address=145.79.227.10:4402 , piece_id=NRS5RYXJWVNHZLB3CP72YKK5KA72OPJAALGHSPQ5Z72QVEM3U7SA, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.119650006 (586h17m7.990733041s) [noise=true , address=145.79.227.39:4013 , piece_id=UA666QW4AWRPUFHTOXYRPOI6CVHJWXIXRV3WRWFSHIATV2GJTJTQ, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.124191873 (586h17m7.995274908s) [noise=true , address=145.79.227.45:4229 , piece_id=VSCFYPE7OQL2RFLZEODFG64KVKKVDP5P27S3XEVHI3JQ3MVA3V3Q, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.258227529 (586h17m8.129310574s) [noise=true , address=77.90.31.14:4662 , piece_id=PKH6TRZOMMDKQ4VRL4WJJYVSANEBUWIAC7DWMOIAKHGONJRICG6A, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.331399385 (586h17m8.20248243s) [noise=true , address=45.91.116.10:4083 , piece_id=5PD42BVDY7LKYZ52I4YZUTKW43TFJLRIM75IFUUPAVTXCV2APFYQ, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.366340226 (586h17m8.237423271s) [noise=true , address=45.91.116.11:4119 , piece_id=SEKS4JREG6VTX25AFMZO43SZXLTNIKP6JUR26OHZWXIM6PEVCMUA, seg_pos={0 0}, upload=513956]: Done uploading piece. err: <nil> Dec 30 01:46:08.366367027 (586h17m8.237450062s) [seg_pos={0 0}, upload=513956]: Segment reached optimal threshold of 13 pieces. Dec 30 01:46:08.366555976 (586h17m8.237639011s) [noise=true , address=77.90.31.13:4629 , piece_id=OWCLLN5DLJ7HG7MIARH4Y74OHHU3DQ4KHHUZOT3LCCYYN4PZHDTQ, seg_pos={0 0}, upload=513956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 01:46:08.366563020 (586h17m8.237646055s) [noise=true , address=77.90.31.13:4629 , piece_id=OWCLLN5DLJ7HG7MIARH4Y74OHHU3DQ4KHHUZOT3LCCYYN4PZHDTQ, seg_pos={0 0}, upload=513956]: Long tail cancellation due to optimal threshold Dec 30 01:46:08.366685844 (586h17m8.237768879s) [noise=true , address=45.91.116.14:4192 , piece_id=37V7RQYB7S6DFSW7PEOBU6BXX5EQ4IP7O53TT5WTECGJ4JJKUOCA, seg_pos={0 0}, upload=513956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 01:46:08.366692236 (586h17m8.237775271s) [noise=true , address=45.91.116.14:4192 , piece_id=37V7RQYB7S6DFSW7PEOBU6BXX5EQ4IP7O53TT5WTECGJ4JJKUOCA, seg_pos={0 0}, upload=513956]: Long tail cancellation due to optimal threshold Dec 30 01:46:08.366764864 (586h17m8.237847939s) [noise=true , address=77.90.31.18:4805 , piece_id=VE7HSMAWGMAXPSRUYGSGYR27N4Q273A4UYIK5WSLTWGSDM57CHOQ, seg_pos={0 0}, upload=513956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:59406->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:59406->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59406->77.90.31.18:4805: use of closed network connection Dec 30 01:46:08.366798158 (586h17m8.237881213s) [noise=true , address=77.90.31.18:4805 , piece_id=VE7HSMAWGMAXPSRUYGSGYR27N4Q273A4UYIK5WSLTWGSDM57CHOQ, seg_pos={0 0}, upload=513956]: Long tail cancellation due to optimal threshold Dec 30 01:46:08.366849114 (586h17m8.237932149s) [seg_pos={0 0}, upload=513956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:46:08.366957411 (586h17m8.238040456s) [upload=513956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:46:08.366963082 (586h17m8.238046127s) [upload=513956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:46:08.654712283 (586h17m8.525795338s) [upload=513956]: Done uploading segments... Dec 30 01:46:08.654737410 (586h17m8.525820445s) [upload=513956]: Upload finished. err: <nil> Duration: 30.003059119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 26.433330701s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:45:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 26.433365556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X.....X.................XX.....X.X......X.X..XX.X....X..] duration: 5.960093634s trace: https://jaeger.dev.storj.io/trace/43626b164577d27d FAIL (2025-12-30 01:44:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA) context deadline exceeded ======LOGS====== Dec 30 01:43:54.069760407 (586h14m53.940843452s) [upload=513924]: Starting upload Dec 30 01:43:54.069940359 (586h14m53.941023404s) [upload=513924]: Done starting upload Dec 30 01:43:54.070030180 (586h14m53.941113215s) [upload=513924]: Uploading segments... Dec 30 01:43:54.070198230 (586h14m53.941281275s) [upload=513924]: Got next segment. Inline: false Dec 30 01:43:54.070262532 (586h14m53.941345567s) [upload=513924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:43:54.070269876 (586h14m53.941352921s) [upload=513924]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:43:54.318212713 (586h14m54.189295768s) [seg_pos={0 0}, upload=513924]: Begin upload segment... Dec 30 01:43:54.318280091 (586h14m54.189363136s) [seg_pos={0 0}, upload=513924]: Done begin upload segment. Dec 30 01:43:54.318299197 (586h14m54.189382242s) [noise=true , address=145.79.227.39:4015 , piece_id=QSK7YP5SKXQAFWZK6FBURPPUJGVETQRGDCPMNN6IAKZSZSBZVDOA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.318319816 (586h14m54.189402862s) [noise=true , address=145.79.227.7:4295 , piece_id=UYU7CFPBXHJUYCGJWVHIK5D5A5NB47PQJ2CBVLCQSAN35GQ75CQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.318406321 (586h14m54.189489356s) [noise=true , address=45.91.116.4:4940 , piece_id=BJLJWUZXJQZPAJCO5ZEFVG24TBVCLRWI44DIQSKGTHYSBRDIL5OA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319100042 (586h14m54.190183087s) [noise=true , address=145.79.227.11:4732 , piece_id=HYLZW3O4QBBIOSA54RNPXEWZ5W6CHLZF5IOJKRJI7OSAQWHOHWQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319498912 (586h14m54.190581947s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319889315 (586h14m54.19097237s) [noise=true , address=145.79.227.45:4231 , piece_id=PGX2U3ZIIAJDFERJPSIQAQ5P5JDLPT7777UDJUFE5LU465HFXVMA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.320289947 (586h14m54.191372992s) [noise=true , address=45.91.116.5:4297 , piece_id=JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.320687826 (586h14m54.191770871s) [noise=true , address=145.79.227.14:4839 , piece_id=U2CCNBF25BMP7FKAUCOAV5Q4YW2W3FYLDZ3J4EXJMNXTLQQXHFMQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.321150306 (586h14m54.192233351s) [noise=true , address=145.79.227.19:4079 , piece_id=THLZFTIXEXFCQFLDC63CAKF3IBU5GHGGOSFCD67XOM2OCMRSSWQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.321608057 (586h14m54.192691112s) [noise=true , address=145.79.227.36:4519 , piece_id=DPGOLWJJA4AU33VNMFT7KB6LT4ZCGJJ4Y7OPJLKNAIT3QROIQGVQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.323232601 (586h14m54.194315636s) [noise=true , address=45.91.116.12:4227 , piece_id=2GUWO2SOG5U5QWS7W7FGBBNZAKQOKPXVIBHU2W6ZJZNTU7YKF76Q, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.323649455 (586h14m54.19473249s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.324627487 (586h14m54.195710532s) [noise=true , address=145.79.227.47:4266 , piece_id=DBX2P36NFP5OHAATI6PTVF6X47QIN5N447MUKICAZLVBZ2DHKVDA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.326992270 (586h14m54.198075305s) [noise=true , address=145.79.227.22:4510 , piece_id=53X5KEAWRBGYBAIR43Q3LHXH4DB5OZ36474PEMFNPHUEMPS4UNIA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.327373165 (586h14m54.19845621s) [noise=true , address=77.90.31.19:4844 , piece_id=FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.331524168 (586h14m54.202607214s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:56.028744110 (586h14m55.899827145s) [upload=513924]: Next returned nil segment Dec 30 01:43:56.028757376 (586h14m55.899840411s) [upload=513924]: Waiting for error group managing segments... Dec 30 01:43:56.309245711 (586h14m56.180328757s) [noise=true , address=145.79.227.19:4079 , piece_id=THLZFTIXEXFCQFLDC63CAKF3IBU5GHGGOSFCD67XOM2OCMRSSWQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.312124963 (586h14m56.183207998s) [noise=true , address=145.79.227.14:4839 , piece_id=U2CCNBF25BMP7FKAUCOAV5Q4YW2W3FYLDZ3J4EXJMNXTLQQXHFMQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.313240858 (586h14m56.184323893s) [noise=true , address=145.79.227.11:4732 , piece_id=HYLZW3O4QBBIOSA54RNPXEWZ5W6CHLZF5IOJKRJI7OSAQWHOHWQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.314964049 (586h14m56.186047084s) [noise=true , address=145.79.227.45:4231 , piece_id=PGX2U3ZIIAJDFERJPSIQAQ5P5JDLPT7777UDJUFE5LU465HFXVMA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.316725574 (586h14m56.187808609s) [noise=true , address=145.79.227.7:4295 , piece_id=UYU7CFPBXHJUYCGJWVHIK5D5A5NB47PQJ2CBVLCQSAN35GQ75CQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.320230597 (586h14m56.191313632s) [noise=true , address=145.79.227.36:4519 , piece_id=DPGOLWJJA4AU33VNMFT7KB6LT4ZCGJJ4Y7OPJLKNAIT3QROIQGVQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.332567377 (586h14m56.203650422s) [noise=true , address=145.79.227.22:4510 , piece_id=53X5KEAWRBGYBAIR43Q3LHXH4DB5OZ36474PEMFNPHUEMPS4UNIA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.336725686 (586h14m56.207808721s) [noise=true , address=145.79.227.39:4015 , piece_id=QSK7YP5SKXQAFWZK6FBURPPUJGVETQRGDCPMNN6IAKZSZSBZVDOA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.347522754 (586h14m56.218605789s) [noise=true , address=145.79.227.47:4266 , piece_id=DBX2P36NFP5OHAATI6PTVF6X47QIN5N447MUKICAZLVBZ2DHKVDA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.610138748 (586h14m56.481221793s) [noise=true , address=77.90.31.19:4844 , piece_id=FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.668717048 (586h14m56.539800083s) [noise=true , address=45.91.116.4:4940 , piece_id=BJLJWUZXJQZPAJCO5ZEFVG24TBVCLRWI44DIQSKGTHYSBRDIL5OA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.711847613 (586h14m56.582930648s) [noise=true , address=45.91.116.12:4227 , piece_id=2GUWO2SOG5U5QWS7W7FGBBNZAKQOKPXVIBHU2W6ZJZNTU7YKF76Q, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:57.282134079 (586h14m57.153217124s) [noise=true , address=45.91.116.5:4297 , piece_id=JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:57.282154719 (586h14m57.153237764s) [seg_pos={0 0}, upload=513924]: Segment reached optimal threshold of 13 pieces. Dec 30 01:43:57.282613132 (586h14m57.153696177s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.282625304 (586h14m57.153708339s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.282793214 (586h14m57.153876249s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.282799155 (586h14m57.153882201s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.283335828 (586h14m57.154418873s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.283343502 (586h14m57.154426537s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.283364272 (586h14m57.154447307s) [seg_pos={0 0}, upload=513924]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:43:57.283428244 (586h14m57.154511279s) [upload=513924]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:43:57.283432983 (586h14m57.154516018s) [upload=513924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:43:57.581905793 (586h14m57.452988838s) [upload=513924]: Done uploading segments... Dec 30 01:43:57.581927885 (586h14m57.45301092s) [upload=513924]: Upload finished. err: <nil> Duration: 5.960121498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X.....X.................XX.....X.X......X.X..XX.X....X.] duration: 6.124607016s trace: https://jaeger.dev.storj.io/trace/43626b164577d27d FAIL (2025-12-30 01:44:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA) context deadline exceeded ======LOGS====== Dec 30 01:43:54.069760407 (586h14m53.940843452s) [upload=513924]: Starting upload Dec 30 01:43:54.069940359 (586h14m53.941023404s) [upload=513924]: Done starting upload Dec 30 01:43:54.070030180 (586h14m53.941113215s) [upload=513924]: Uploading segments... Dec 30 01:43:54.070198230 (586h14m53.941281275s) [upload=513924]: Got next segment. Inline: false Dec 30 01:43:54.070262532 (586h14m53.941345567s) [upload=513924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:43:54.070269876 (586h14m53.941352921s) [upload=513924]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:43:54.318212713 (586h14m54.189295768s) [seg_pos={0 0}, upload=513924]: Begin upload segment... Dec 30 01:43:54.318280091 (586h14m54.189363136s) [seg_pos={0 0}, upload=513924]: Done begin upload segment. Dec 30 01:43:54.318299197 (586h14m54.189382242s) [noise=true , address=145.79.227.39:4015 , piece_id=QSK7YP5SKXQAFWZK6FBURPPUJGVETQRGDCPMNN6IAKZSZSBZVDOA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.318319816 (586h14m54.189402862s) [noise=true , address=145.79.227.7:4295 , piece_id=UYU7CFPBXHJUYCGJWVHIK5D5A5NB47PQJ2CBVLCQSAN35GQ75CQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.318406321 (586h14m54.189489356s) [noise=true , address=45.91.116.4:4940 , piece_id=BJLJWUZXJQZPAJCO5ZEFVG24TBVCLRWI44DIQSKGTHYSBRDIL5OA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319100042 (586h14m54.190183087s) [noise=true , address=145.79.227.11:4732 , piece_id=HYLZW3O4QBBIOSA54RNPXEWZ5W6CHLZF5IOJKRJI7OSAQWHOHWQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319498912 (586h14m54.190581947s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319889315 (586h14m54.19097237s) [noise=true , address=145.79.227.45:4231 , piece_id=PGX2U3ZIIAJDFERJPSIQAQ5P5JDLPT7777UDJUFE5LU465HFXVMA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.320289947 (586h14m54.191372992s) [noise=true , address=45.91.116.5:4297 , piece_id=JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.320687826 (586h14m54.191770871s) [noise=true , address=145.79.227.14:4839 , piece_id=U2CCNBF25BMP7FKAUCOAV5Q4YW2W3FYLDZ3J4EXJMNXTLQQXHFMQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.321150306 (586h14m54.192233351s) [noise=true , address=145.79.227.19:4079 , piece_id=THLZFTIXEXFCQFLDC63CAKF3IBU5GHGGOSFCD67XOM2OCMRSSWQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.321608057 (586h14m54.192691112s) [noise=true , address=145.79.227.36:4519 , piece_id=DPGOLWJJA4AU33VNMFT7KB6LT4ZCGJJ4Y7OPJLKNAIT3QROIQGVQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.323232601 (586h14m54.194315636s) [noise=true , address=45.91.116.12:4227 , piece_id=2GUWO2SOG5U5QWS7W7FGBBNZAKQOKPXVIBHU2W6ZJZNTU7YKF76Q, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.323649455 (586h14m54.19473249s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.324627487 (586h14m54.195710532s) [noise=true , address=145.79.227.47:4266 , piece_id=DBX2P36NFP5OHAATI6PTVF6X47QIN5N447MUKICAZLVBZ2DHKVDA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.326992270 (586h14m54.198075305s) [noise=true , address=145.79.227.22:4510 , piece_id=53X5KEAWRBGYBAIR43Q3LHXH4DB5OZ36474PEMFNPHUEMPS4UNIA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.327373165 (586h14m54.19845621s) [noise=true , address=77.90.31.19:4844 , piece_id=FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.331524168 (586h14m54.202607214s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:56.028744110 (586h14m55.899827145s) [upload=513924]: Next returned nil segment Dec 30 01:43:56.028757376 (586h14m55.899840411s) [upload=513924]: Waiting for error group managing segments... Dec 30 01:43:56.309245711 (586h14m56.180328757s) [noise=true , address=145.79.227.19:4079 , piece_id=THLZFTIXEXFCQFLDC63CAKF3IBU5GHGGOSFCD67XOM2OCMRSSWQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.312124963 (586h14m56.183207998s) [noise=true , address=145.79.227.14:4839 , piece_id=U2CCNBF25BMP7FKAUCOAV5Q4YW2W3FYLDZ3J4EXJMNXTLQQXHFMQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.313240858 (586h14m56.184323893s) [noise=true , address=145.79.227.11:4732 , piece_id=HYLZW3O4QBBIOSA54RNPXEWZ5W6CHLZF5IOJKRJI7OSAQWHOHWQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.314964049 (586h14m56.186047084s) [noise=true , address=145.79.227.45:4231 , piece_id=PGX2U3ZIIAJDFERJPSIQAQ5P5JDLPT7777UDJUFE5LU465HFXVMA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.316725574 (586h14m56.187808609s) [noise=true , address=145.79.227.7:4295 , piece_id=UYU7CFPBXHJUYCGJWVHIK5D5A5NB47PQJ2CBVLCQSAN35GQ75CQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.320230597 (586h14m56.191313632s) [noise=true , address=145.79.227.36:4519 , piece_id=DPGOLWJJA4AU33VNMFT7KB6LT4ZCGJJ4Y7OPJLKNAIT3QROIQGVQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.332567377 (586h14m56.203650422s) [noise=true , address=145.79.227.22:4510 , piece_id=53X5KEAWRBGYBAIR43Q3LHXH4DB5OZ36474PEMFNPHUEMPS4UNIA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.336725686 (586h14m56.207808721s) [noise=true , address=145.79.227.39:4015 , piece_id=QSK7YP5SKXQAFWZK6FBURPPUJGVETQRGDCPMNN6IAKZSZSBZVDOA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.347522754 (586h14m56.218605789s) [noise=true , address=145.79.227.47:4266 , piece_id=DBX2P36NFP5OHAATI6PTVF6X47QIN5N447MUKICAZLVBZ2DHKVDA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.610138748 (586h14m56.481221793s) [noise=true , address=77.90.31.19:4844 , piece_id=FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.668717048 (586h14m56.539800083s) [noise=true , address=45.91.116.4:4940 , piece_id=BJLJWUZXJQZPAJCO5ZEFVG24TBVCLRWI44DIQSKGTHYSBRDIL5OA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.711847613 (586h14m56.582930648s) [noise=true , address=45.91.116.12:4227 , piece_id=2GUWO2SOG5U5QWS7W7FGBBNZAKQOKPXVIBHU2W6ZJZNTU7YKF76Q, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:57.282134079 (586h14m57.153217124s) [noise=true , address=45.91.116.5:4297 , piece_id=JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:57.282154719 (586h14m57.153237764s) [seg_pos={0 0}, upload=513924]: Segment reached optimal threshold of 13 pieces. Dec 30 01:43:57.282613132 (586h14m57.153696177s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.282625304 (586h14m57.153708339s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.282793214 (586h14m57.153876249s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.282799155 (586h14m57.153882201s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.283335828 (586h14m57.154418873s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.283343502 (586h14m57.154426537s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.283364272 (586h14m57.154447307s) [seg_pos={0 0}, upload=513924]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:43:57.283428244 (586h14m57.154511279s) [upload=513924]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:43:57.283432983 (586h14m57.154516018s) [upload=513924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:43:57.581905793 (586h14m57.452988838s) [upload=513924]: Done uploading segments... Dec 30 01:43:57.581927885 (586h14m57.45301092s) [upload=513924]: Upload finished. err: <nil> Duration: 6.124628828s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.X.....X.................XX.....X.X......X.X..XX.X....X] duration: 30.002911815s trace: https://jaeger.dev.storj.io/trace/43626b164577d27d FAIL (2025-12-30 01:44:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA) context deadline exceeded ======LOGS====== Dec 30 01:43:54.069760407 (586h14m53.940843452s) [upload=513924]: Starting upload Dec 30 01:43:54.069940359 (586h14m53.941023404s) [upload=513924]: Done starting upload Dec 30 01:43:54.070030180 (586h14m53.941113215s) [upload=513924]: Uploading segments... Dec 30 01:43:54.070198230 (586h14m53.941281275s) [upload=513924]: Got next segment. Inline: false Dec 30 01:43:54.070262532 (586h14m53.941345567s) [upload=513924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:43:54.070269876 (586h14m53.941352921s) [upload=513924]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:43:54.318212713 (586h14m54.189295768s) [seg_pos={0 0}, upload=513924]: Begin upload segment... Dec 30 01:43:54.318280091 (586h14m54.189363136s) [seg_pos={0 0}, upload=513924]: Done begin upload segment. Dec 30 01:43:54.318299197 (586h14m54.189382242s) [noise=true , address=145.79.227.39:4015 , piece_id=QSK7YP5SKXQAFWZK6FBURPPUJGVETQRGDCPMNN6IAKZSZSBZVDOA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.318319816 (586h14m54.189402862s) [noise=true , address=145.79.227.7:4295 , piece_id=UYU7CFPBXHJUYCGJWVHIK5D5A5NB47PQJ2CBVLCQSAN35GQ75CQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.318406321 (586h14m54.189489356s) [noise=true , address=45.91.116.4:4940 , piece_id=BJLJWUZXJQZPAJCO5ZEFVG24TBVCLRWI44DIQSKGTHYSBRDIL5OA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319100042 (586h14m54.190183087s) [noise=true , address=145.79.227.11:4732 , piece_id=HYLZW3O4QBBIOSA54RNPXEWZ5W6CHLZF5IOJKRJI7OSAQWHOHWQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319498912 (586h14m54.190581947s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.319889315 (586h14m54.19097237s) [noise=true , address=145.79.227.45:4231 , piece_id=PGX2U3ZIIAJDFERJPSIQAQ5P5JDLPT7777UDJUFE5LU465HFXVMA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.320289947 (586h14m54.191372992s) [noise=true , address=45.91.116.5:4297 , piece_id=JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.320687826 (586h14m54.191770871s) [noise=true , address=145.79.227.14:4839 , piece_id=U2CCNBF25BMP7FKAUCOAV5Q4YW2W3FYLDZ3J4EXJMNXTLQQXHFMQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.321150306 (586h14m54.192233351s) [noise=true , address=145.79.227.19:4079 , piece_id=THLZFTIXEXFCQFLDC63CAKF3IBU5GHGGOSFCD67XOM2OCMRSSWQQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.321608057 (586h14m54.192691112s) [noise=true , address=145.79.227.36:4519 , piece_id=DPGOLWJJA4AU33VNMFT7KB6LT4ZCGJJ4Y7OPJLKNAIT3QROIQGVQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.323232601 (586h14m54.194315636s) [noise=true , address=45.91.116.12:4227 , piece_id=2GUWO2SOG5U5QWS7W7FGBBNZAKQOKPXVIBHU2W6ZJZNTU7YKF76Q, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.323649455 (586h14m54.19473249s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.324627487 (586h14m54.195710532s) [noise=true , address=145.79.227.47:4266 , piece_id=DBX2P36NFP5OHAATI6PTVF6X47QIN5N447MUKICAZLVBZ2DHKVDA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.326992270 (586h14m54.198075305s) [noise=true , address=145.79.227.22:4510 , piece_id=53X5KEAWRBGYBAIR43Q3LHXH4DB5OZ36474PEMFNPHUEMPS4UNIA, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.327373165 (586h14m54.19845621s) [noise=true , address=77.90.31.19:4844 , piece_id=FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:54.331524168 (586h14m54.202607214s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Uploading piece... Dec 30 01:43:56.028744110 (586h14m55.899827145s) [upload=513924]: Next returned nil segment Dec 30 01:43:56.028757376 (586h14m55.899840411s) [upload=513924]: Waiting for error group managing segments... Dec 30 01:43:56.309245711 (586h14m56.180328757s) [noise=true , address=145.79.227.19:4079 , piece_id=THLZFTIXEXFCQFLDC63CAKF3IBU5GHGGOSFCD67XOM2OCMRSSWQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.312124963 (586h14m56.183207998s) [noise=true , address=145.79.227.14:4839 , piece_id=U2CCNBF25BMP7FKAUCOAV5Q4YW2W3FYLDZ3J4EXJMNXTLQQXHFMQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.313240858 (586h14m56.184323893s) [noise=true , address=145.79.227.11:4732 , piece_id=HYLZW3O4QBBIOSA54RNPXEWZ5W6CHLZF5IOJKRJI7OSAQWHOHWQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.314964049 (586h14m56.186047084s) [noise=true , address=145.79.227.45:4231 , piece_id=PGX2U3ZIIAJDFERJPSIQAQ5P5JDLPT7777UDJUFE5LU465HFXVMA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.316725574 (586h14m56.187808609s) [noise=true , address=145.79.227.7:4295 , piece_id=UYU7CFPBXHJUYCGJWVHIK5D5A5NB47PQJ2CBVLCQSAN35GQ75CQQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.320230597 (586h14m56.191313632s) [noise=true , address=145.79.227.36:4519 , piece_id=DPGOLWJJA4AU33VNMFT7KB6LT4ZCGJJ4Y7OPJLKNAIT3QROIQGVQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.332567377 (586h14m56.203650422s) [noise=true , address=145.79.227.22:4510 , piece_id=53X5KEAWRBGYBAIR43Q3LHXH4DB5OZ36474PEMFNPHUEMPS4UNIA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.336725686 (586h14m56.207808721s) [noise=true , address=145.79.227.39:4015 , piece_id=QSK7YP5SKXQAFWZK6FBURPPUJGVETQRGDCPMNN6IAKZSZSBZVDOA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.347522754 (586h14m56.218605789s) [noise=true , address=145.79.227.47:4266 , piece_id=DBX2P36NFP5OHAATI6PTVF6X47QIN5N447MUKICAZLVBZ2DHKVDA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.610138748 (586h14m56.481221793s) [noise=true , address=77.90.31.19:4844 , piece_id=FAUQRE77JIYFRJZGKEJ2ZETCG4CQ2N5KGI6GC6HDGCTZCR2FZD3A, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.668717048 (586h14m56.539800083s) [noise=true , address=45.91.116.4:4940 , piece_id=BJLJWUZXJQZPAJCO5ZEFVG24TBVCLRWI44DIQSKGTHYSBRDIL5OA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:56.711847613 (586h14m56.582930648s) [noise=true , address=45.91.116.12:4227 , piece_id=2GUWO2SOG5U5QWS7W7FGBBNZAKQOKPXVIBHU2W6ZJZNTU7YKF76Q, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:57.282134079 (586h14m57.153217124s) [noise=true , address=45.91.116.5:4297 , piece_id=JDYMMS6SBBCJDTF6P5UIHNRRCF7B4PAZDL26NCOO4KTLM66NNXXA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: <nil> Dec 30 01:43:57.282154719 (586h14m57.153237764s) [seg_pos={0 0}, upload=513924]: Segment reached optimal threshold of 13 pieces. Dec 30 01:43:57.282613132 (586h14m57.153696177s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.282625304 (586h14m57.153708339s) [noise=true , address=145.79.227.49:4339 , piece_id=6EP5SZXTT3F2AJYJWQ3GZJCDFYUPOUJ2PXFWK74GDGSLMOMBWCIQ, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.282793214 (586h14m57.153876249s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.282799155 (586h14m57.153882201s) [noise=true , address=145.79.227.44:4195 , piece_id=74MC7YWSDLFLM26V4UXW2Y3V3NDFENIDGA7HY4UAYSDZLLZILKIQ, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.283335828 (586h14m57.154418873s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:43:57.283343502 (586h14m57.154426537s) [noise=true , address=145.79.227.35:4481 , piece_id=CYXJ4YKKMWJMZKAKB7VO2RUVS5PY6MXMKFQD3CA5VQECZRGEYMUA, seg_pos={0 0}, upload=513924]: Long tail cancellation due to optimal threshold Dec 30 01:43:57.283364272 (586h14m57.154447307s) [seg_pos={0 0}, upload=513924]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:43:57.283428244 (586h14m57.154511279s) [upload=513924]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:43:57.283432983 (586h14m57.154516018s) [upload=513924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:43:57.581905793 (586h14m57.452988838s) [upload=513924]: Done uploading segments... Dec 30 01:43:57.581927885 (586h14m57.45301092s) [upload=513924]: Upload finished. err: <nil> Duration: 30.002933055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 14.825713443s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:43:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 14.825728362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 35.202939331s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:43:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 35.202967987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....X.X.....X.................XX.....X.X......X.X..XX.X....] duration: 5.750933455s trace: https://jaeger.dev.storj.io/trace/38d2949eae436bc8 FAIL (2025-12-30 01:42:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:39:41.242896830 (586h10m41.113979875s) [upload=513865]: Starting upload Dec 30 01:39:41.242962234 (586h10m41.114045279s) [upload=513865]: Done starting upload Dec 30 01:39:41.243011488 (586h10m41.114094533s) [upload=513865]: Uploading segments... Dec 30 01:39:41.243158889 (586h10m41.114241924s) [upload=513865]: Got next segment. Inline: false Dec 30 01:39:41.243199727 (586h10m41.114282762s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:39:41.243204225 (586h10m41.11428726s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:39:41.492477874 (586h10m41.363560909s) [seg_pos={0 0}, upload=513865]: Begin upload segment... Dec 30 01:39:41.492555182 (586h10m41.363638227s) [seg_pos={0 0}, upload=513865]: Done begin upload segment. Dec 30 01:39:41.492614985 (586h10m41.36369803s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.492706890 (586h10m41.363789935s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493132370 (586h10m41.364215415s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493443653 (586h10m41.364526698s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493721913 (586h10m41.364804948s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494297289 (586h10m41.365380334s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494735953 (586h10m41.365818988s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495061864 (586h10m41.366144919s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495398224 (586h10m41.36648127s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495730628 (586h10m41.366813673s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496582850 (586h10m41.367665885s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496862082 (586h10m41.367945117s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497929595 (586h10m41.36901263s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497999578 (586h10m41.369082613s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499108208 (586h10m41.370191253s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499474697 (586h10m41.370557732s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:43.188028760 (586h10m43.059111795s) [upload=513865]: Next returned nil segment Dec 30 01:39:43.188064568 (586h10m43.059147613s) [upload=513865]: Waiting for error group managing segments... Dec 30 01:39:43.467597093 (586h10m43.338680138s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.487657630 (586h10m43.358740676s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488377821 (586h10m43.359460866s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488907601 (586h10m43.359990656s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.489478217 (586h10m43.360561262s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490129868 (586h10m43.361212913s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490198439 (586h10m43.361281494s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.494357257 (586h10m43.365440302s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.495001975 (586h10m43.36608503s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.505726265 (586h10m43.37680932s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.682758881 (586h10m43.553841916s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.692514777 (586h10m43.563597832s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695753394 (586h10m43.566836439s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695773432 (586h10m43.566856467s) [seg_pos={0 0}, upload=513865]: Segment reached optimal threshold of 13 pieces. Dec 30 01:39:43.695946813 (586h10m43.567029848s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.695953315 (586h10m43.56703635s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696191559 (586h10m43.567274604s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696201207 (586h10m43.567284242s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696509814 (586h10m43.56759286s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696530543 (586h10m43.567613598s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696585569 (586h10m43.567668624s) [seg_pos={0 0}, upload=513865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:39:43.696706289 (586h10m43.567789334s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:39:43.696713222 (586h10m43.567796267s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:39:43.983467418 (586h10m43.854550453s) [upload=513865]: Done uploading segments... Dec 30 01:39:43.983485271 (586h10m43.854568306s) [upload=513865]: Upload finished. err: <nil> Duration: 5.750960467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 25.286805334s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 25.286832115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X....X.X.....X.................XX.....X.X......X.X..XX.X...] duration: 5.989881653s trace: https://jaeger.dev.storj.io/trace/38d2949eae436bc8 FAIL (2025-12-30 01:41:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:39:41.242896830 (586h10m41.113979875s) [upload=513865]: Starting upload Dec 30 01:39:41.242962234 (586h10m41.114045279s) [upload=513865]: Done starting upload Dec 30 01:39:41.243011488 (586h10m41.114094533s) [upload=513865]: Uploading segments... Dec 30 01:39:41.243158889 (586h10m41.114241924s) [upload=513865]: Got next segment. Inline: false Dec 30 01:39:41.243199727 (586h10m41.114282762s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:39:41.243204225 (586h10m41.11428726s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:39:41.492477874 (586h10m41.363560909s) [seg_pos={0 0}, upload=513865]: Begin upload segment... Dec 30 01:39:41.492555182 (586h10m41.363638227s) [seg_pos={0 0}, upload=513865]: Done begin upload segment. Dec 30 01:39:41.492614985 (586h10m41.36369803s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.492706890 (586h10m41.363789935s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493132370 (586h10m41.364215415s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493443653 (586h10m41.364526698s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493721913 (586h10m41.364804948s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494297289 (586h10m41.365380334s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494735953 (586h10m41.365818988s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495061864 (586h10m41.366144919s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495398224 (586h10m41.36648127s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495730628 (586h10m41.366813673s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496582850 (586h10m41.367665885s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496862082 (586h10m41.367945117s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497929595 (586h10m41.36901263s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497999578 (586h10m41.369082613s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499108208 (586h10m41.370191253s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499474697 (586h10m41.370557732s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:43.188028760 (586h10m43.059111795s) [upload=513865]: Next returned nil segment Dec 30 01:39:43.188064568 (586h10m43.059147613s) [upload=513865]: Waiting for error group managing segments... Dec 30 01:39:43.467597093 (586h10m43.338680138s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.487657630 (586h10m43.358740676s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488377821 (586h10m43.359460866s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488907601 (586h10m43.359990656s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.489478217 (586h10m43.360561262s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490129868 (586h10m43.361212913s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490198439 (586h10m43.361281494s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.494357257 (586h10m43.365440302s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.495001975 (586h10m43.36608503s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.505726265 (586h10m43.37680932s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.682758881 (586h10m43.553841916s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.692514777 (586h10m43.563597832s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695753394 (586h10m43.566836439s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695773432 (586h10m43.566856467s) [seg_pos={0 0}, upload=513865]: Segment reached optimal threshold of 13 pieces. Dec 30 01:39:43.695946813 (586h10m43.567029848s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.695953315 (586h10m43.56703635s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696191559 (586h10m43.567274604s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696201207 (586h10m43.567284242s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696509814 (586h10m43.56759286s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696530543 (586h10m43.567613598s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696585569 (586h10m43.567668624s) [seg_pos={0 0}, upload=513865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:39:43.696706289 (586h10m43.567789334s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:39:43.696713222 (586h10m43.567796267s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:39:43.983467418 (586h10m43.854550453s) [upload=513865]: Done uploading segments... Dec 30 01:39:43.983485271 (586h10m43.854568306s) [upload=513865]: Upload finished. err: <nil> Duration: 5.989910396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X....X.X.....X.................XX.....X.X......X.X..XX.X..] duration: 11.205901596s trace: https://jaeger.dev.storj.io/trace/38d2949eae436bc8 FAIL (2025-12-30 01:41:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:39:41.242896830 (586h10m41.113979875s) [upload=513865]: Starting upload Dec 30 01:39:41.242962234 (586h10m41.114045279s) [upload=513865]: Done starting upload Dec 30 01:39:41.243011488 (586h10m41.114094533s) [upload=513865]: Uploading segments... Dec 30 01:39:41.243158889 (586h10m41.114241924s) [upload=513865]: Got next segment. Inline: false Dec 30 01:39:41.243199727 (586h10m41.114282762s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:39:41.243204225 (586h10m41.11428726s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:39:41.492477874 (586h10m41.363560909s) [seg_pos={0 0}, upload=513865]: Begin upload segment... Dec 30 01:39:41.492555182 (586h10m41.363638227s) [seg_pos={0 0}, upload=513865]: Done begin upload segment. Dec 30 01:39:41.492614985 (586h10m41.36369803s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.492706890 (586h10m41.363789935s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493132370 (586h10m41.364215415s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493443653 (586h10m41.364526698s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493721913 (586h10m41.364804948s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494297289 (586h10m41.365380334s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494735953 (586h10m41.365818988s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495061864 (586h10m41.366144919s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495398224 (586h10m41.36648127s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495730628 (586h10m41.366813673s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496582850 (586h10m41.367665885s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496862082 (586h10m41.367945117s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497929595 (586h10m41.36901263s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497999578 (586h10m41.369082613s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499108208 (586h10m41.370191253s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499474697 (586h10m41.370557732s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:43.188028760 (586h10m43.059111795s) [upload=513865]: Next returned nil segment Dec 30 01:39:43.188064568 (586h10m43.059147613s) [upload=513865]: Waiting for error group managing segments... Dec 30 01:39:43.467597093 (586h10m43.338680138s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.487657630 (586h10m43.358740676s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488377821 (586h10m43.359460866s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488907601 (586h10m43.359990656s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.489478217 (586h10m43.360561262s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490129868 (586h10m43.361212913s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490198439 (586h10m43.361281494s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.494357257 (586h10m43.365440302s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.495001975 (586h10m43.36608503s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.505726265 (586h10m43.37680932s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.682758881 (586h10m43.553841916s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.692514777 (586h10m43.563597832s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695753394 (586h10m43.566836439s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695773432 (586h10m43.566856467s) [seg_pos={0 0}, upload=513865]: Segment reached optimal threshold of 13 pieces. Dec 30 01:39:43.695946813 (586h10m43.567029848s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.695953315 (586h10m43.56703635s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696191559 (586h10m43.567274604s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696201207 (586h10m43.567284242s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696509814 (586h10m43.56759286s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696530543 (586h10m43.567613598s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696585569 (586h10m43.567668624s) [seg_pos={0 0}, upload=513865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:39:43.696706289 (586h10m43.567789334s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:39:43.696713222 (586h10m43.567796267s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:39:43.983467418 (586h10m43.854550453s) [upload=513865]: Done uploading segments... Dec 30 01:39:43.983485271 (586h10m43.854568306s) [upload=513865]: Upload finished. err: <nil> Duration: 11.205969748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 15.609953739s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:41:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.60998536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 47.612045739s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:40:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 47.612063886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X....X.X.....X.................XX.....X.X......X.X..XX.X.] duration: 6.192960739s trace: https://jaeger.dev.storj.io/trace/38d2949eae436bc8 FAIL (2025-12-30 01:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:39:41.242896830 (586h10m41.113979875s) [upload=513865]: Starting upload Dec 30 01:39:41.242962234 (586h10m41.114045279s) [upload=513865]: Done starting upload Dec 30 01:39:41.243011488 (586h10m41.114094533s) [upload=513865]: Uploading segments... Dec 30 01:39:41.243158889 (586h10m41.114241924s) [upload=513865]: Got next segment. Inline: false Dec 30 01:39:41.243199727 (586h10m41.114282762s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:39:41.243204225 (586h10m41.11428726s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:39:41.492477874 (586h10m41.363560909s) [seg_pos={0 0}, upload=513865]: Begin upload segment... Dec 30 01:39:41.492555182 (586h10m41.363638227s) [seg_pos={0 0}, upload=513865]: Done begin upload segment. Dec 30 01:39:41.492614985 (586h10m41.36369803s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.492706890 (586h10m41.363789935s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493132370 (586h10m41.364215415s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493443653 (586h10m41.364526698s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493721913 (586h10m41.364804948s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494297289 (586h10m41.365380334s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494735953 (586h10m41.365818988s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495061864 (586h10m41.366144919s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495398224 (586h10m41.36648127s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495730628 (586h10m41.366813673s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496582850 (586h10m41.367665885s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496862082 (586h10m41.367945117s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497929595 (586h10m41.36901263s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497999578 (586h10m41.369082613s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499108208 (586h10m41.370191253s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499474697 (586h10m41.370557732s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:43.188028760 (586h10m43.059111795s) [upload=513865]: Next returned nil segment Dec 30 01:39:43.188064568 (586h10m43.059147613s) [upload=513865]: Waiting for error group managing segments... Dec 30 01:39:43.467597093 (586h10m43.338680138s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.487657630 (586h10m43.358740676s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488377821 (586h10m43.359460866s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488907601 (586h10m43.359990656s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.489478217 (586h10m43.360561262s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490129868 (586h10m43.361212913s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490198439 (586h10m43.361281494s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.494357257 (586h10m43.365440302s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.495001975 (586h10m43.36608503s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.505726265 (586h10m43.37680932s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.682758881 (586h10m43.553841916s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.692514777 (586h10m43.563597832s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695753394 (586h10m43.566836439s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695773432 (586h10m43.566856467s) [seg_pos={0 0}, upload=513865]: Segment reached optimal threshold of 13 pieces. Dec 30 01:39:43.695946813 (586h10m43.567029848s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.695953315 (586h10m43.56703635s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696191559 (586h10m43.567274604s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696201207 (586h10m43.567284242s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696509814 (586h10m43.56759286s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696530543 (586h10m43.567613598s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696585569 (586h10m43.567668624s) [seg_pos={0 0}, upload=513865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:39:43.696706289 (586h10m43.567789334s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:39:43.696713222 (586h10m43.567796267s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:39:43.983467418 (586h10m43.854550453s) [upload=513865]: Done uploading segments... Dec 30 01:39:43.983485271 (586h10m43.854568306s) [upload=513865]: Upload finished. err: <nil> Duration: 6.192986837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X....X.X.....X.................XX.....X.X......X.X..XX.X] duration: 30.00314553s trace: https://jaeger.dev.storj.io/trace/38d2949eae436bc8 FAIL (2025-12-30 01:40:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:39:41.242896830 (586h10m41.113979875s) [upload=513865]: Starting upload Dec 30 01:39:41.242962234 (586h10m41.114045279s) [upload=513865]: Done starting upload Dec 30 01:39:41.243011488 (586h10m41.114094533s) [upload=513865]: Uploading segments... Dec 30 01:39:41.243158889 (586h10m41.114241924s) [upload=513865]: Got next segment. Inline: false Dec 30 01:39:41.243199727 (586h10m41.114282762s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:39:41.243204225 (586h10m41.11428726s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:39:41.492477874 (586h10m41.363560909s) [seg_pos={0 0}, upload=513865]: Begin upload segment... Dec 30 01:39:41.492555182 (586h10m41.363638227s) [seg_pos={0 0}, upload=513865]: Done begin upload segment. Dec 30 01:39:41.492614985 (586h10m41.36369803s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.492706890 (586h10m41.363789935s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493132370 (586h10m41.364215415s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493443653 (586h10m41.364526698s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.493721913 (586h10m41.364804948s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494297289 (586h10m41.365380334s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.494735953 (586h10m41.365818988s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495061864 (586h10m41.366144919s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495398224 (586h10m41.36648127s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.495730628 (586h10m41.366813673s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496582850 (586h10m41.367665885s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.496862082 (586h10m41.367945117s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497929595 (586h10m41.36901263s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.497999578 (586h10m41.369082613s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499108208 (586h10m41.370191253s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:41.499474697 (586h10m41.370557732s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Uploading piece... Dec 30 01:39:43.188028760 (586h10m43.059111795s) [upload=513865]: Next returned nil segment Dec 30 01:39:43.188064568 (586h10m43.059147613s) [upload=513865]: Waiting for error group managing segments... Dec 30 01:39:43.467597093 (586h10m43.338680138s) [noise=true , address=145.79.227.7:4294 , piece_id=432TJI35DXUG4VYJ6UR623DGONO6VSYJMWRGGVIMCBP7QVDEO5ZQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.487657630 (586h10m43.358740676s) [noise=true , address=145.79.227.14:4839 , piece_id=EUSNPJFANQRNRFVSGGXTG4ICQUNIRYON5UYYHPZLFYOACFJ5QXRA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488377821 (586h10m43.359460866s) [noise=true , address=145.79.227.11:4731 , piece_id=HI74XMFPENMWL7UINHAJYHLJJIM6FRRN55EVKAJXCWPW7SBDYPPQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.488907601 (586h10m43.359990656s) [noise=true , address=145.79.227.13:4802 , piece_id=AUKHZFRTXG7G4SJJPZEFHER5C44MJTOOBIU4AGJJO5U7N6PHQIYA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.489478217 (586h10m43.360561262s) [noise=true , address=145.79.227.51:4410 , piece_id=BI2QJM2ACLQ3AF4KJFERJSCFMF76OYBT2LKFD3CJPRKF5YUV5WGA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490129868 (586h10m43.361212913s) [noise=true , address=145.79.227.46:4602 , piece_id=QPXXY52E5SCVBEGVTFQYP7W2ZF6AKR7AKBNLDKJB4V2PTNBN3XUA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.490198439 (586h10m43.361281494s) [noise=true , address=145.79.227.52:4445 , piece_id=SNIEDBGHJHOPSPQB6U7ZZIOT2LITAYFUSR43PQP5IO7JB4MHHZOA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.494357257 (586h10m43.365440302s) [noise=true , address=145.79.227.3:4151 , piece_id=AMFAEWHKRTRFWO6XDYBBVXBVM2TO2HPXTBP5Q7CHFXOIYMNNTFGQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.495001975 (586h10m43.36608503s) [noise=true , address=145.79.227.15:4874 , piece_id=C55FWF653UFN6QCMZ6VBKFBFJUAYBPJEUWF22ZN2SGSZ6H2QNVQA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.505726265 (586h10m43.37680932s) [noise=true , address=145.79.227.20:4439 , piece_id=GIFLFKT4QB4VHXQ3NG5WQ5HMMQH4VDTXI32W5SXQENCH4OMOTFUQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.682758881 (586h10m43.553841916s) [noise=true , address=77.90.31.15:4697 , piece_id=4HXXPWDRW2GE5MGKE4F5TEZYHNHW72UTEDGN5FXARRRZFGC22KCQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.692514777 (586h10m43.563597832s) [noise=true , address=77.90.31.17:4770 , piece_id=YHOHVE2BB4AEE6H5A6TSXJ6VKTET7LS25XLPABWZJECAXHMAC6HQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695753394 (586h10m43.566836439s) [noise=true , address=77.90.31.11:4554 , piece_id=KRBBSZHOHKSZIGR32Y3K5GFJCO4LLXR45RRVZGMMRIGSE6VXLH7A, seg_pos={0 0}, upload=513865]: Done uploading piece. err: <nil> Dec 30 01:39:43.695773432 (586h10m43.566856467s) [seg_pos={0 0}, upload=513865]: Segment reached optimal threshold of 13 pieces. Dec 30 01:39:43.695946813 (586h10m43.567029848s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.695953315 (586h10m43.56703635s) [noise=true , address=77.90.31.14:4662 , piece_id=HPEFBIMS7IYDOQLIZVPHGJLQDF4CW4QDFR3PIDJY4H6ZXHVTXZFA, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696191559 (586h10m43.567274604s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696201207 (586h10m43.567284242s) [noise=true , address=77.90.31.19:4842 , piece_id=5TBEMLD7ADIE5ACHRBLL7DIA2TAHTUUSLE4WHEF44U3FIH7C3CXQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696509814 (586h10m43.56759286s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 01:39:43.696530543 (586h10m43.567613598s) [noise=true , address=145.79.227.29:4942 , piece_id=BGWC5VQRUGHXNANYDIGZLQTHZFJZE4M4SYR2GY2XS6PGJSJTLDSQ, seg_pos={0 0}, upload=513865]: Long tail cancellation due to optimal threshold Dec 30 01:39:43.696585569 (586h10m43.567668624s) [seg_pos={0 0}, upload=513865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:39:43.696706289 (586h10m43.567789334s) [upload=513865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:39:43.696713222 (586h10m43.567796267s) [upload=513865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:39:43.983467418 (586h10m43.854550453s) [upload=513865]: Done uploading segments... Dec 30 01:39:43.983485271 (586h10m43.854568306s) [upload=513865]: Upload finished. err: <nil> Duration: 30.003170047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 25.596157426s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:39:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 25.596174549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 31.625888909s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:38:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 31.625919307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X...X....X.X.....X.................XX.....X.X......X.X..XX.] duration: 6.020282881s trace: https://jaeger.dev.storj.io/trace/1b6d44ecfce121e7 FAIL (2025-12-30 01:38:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:37:50.379992278 (586h8m50.251075323s) [upload=513837]: Starting upload Dec 30 01:37:50.380101345 (586h8m50.25118439s) [upload=513837]: Done starting upload Dec 30 01:37:50.380205363 (586h8m50.251288398s) [upload=513837]: Uploading segments... Dec 30 01:37:50.380373043 (586h8m50.251456078s) [upload=513837]: Got next segment. Inline: false Dec 30 01:37:50.380414551 (586h8m50.251497587s) [upload=513837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:37:50.380420613 (586h8m50.251503658s) [upload=513837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:37:50.625426289 (586h8m50.496509324s) [seg_pos={0 0}, upload=513837]: Begin upload segment... Dec 30 01:37:50.625474971 (586h8m50.496558006s) [seg_pos={0 0}, upload=513837]: Done begin upload segment. Dec 30 01:37:50.625499919 (586h8m50.496582954s) [noise=true , address=145.79.227.13:4802 , piece_id=66IYTXYJJQOASFN55SNCRKQ6XTEG7RX3U6KZKL6BYZ3LGSG5SFEA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.625581664 (586h8m50.496664699s) [noise=true , address=45.91.116.7:4012 , piece_id=X4HFHRLPY5N3YESK3FDUV3L7L5ZSAX7HPB34FLZ2VPKJVGPKKOTQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626419940 (586h8m50.497502985s) [noise=true , address=145.79.227.46:4603 , piece_id=BY7QVLNIAIIJYXEZKOIE2HK53WVX5UZAKRNSBBIGUNMMRPXWHY2Q, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626543516 (586h8m50.497626561s) [noise=true , address=145.79.227.19:4079 , piece_id=5EAL2NGIHFXMYBCBHYJHX5CQC6LCALTAXH5ZVHIKAWSX7BEPLJMQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626725733 (586h8m50.497808768s) [noise=true , address=145.79.227.38:4977 , piece_id=T2HZLMIQZE5GS6ZJ6V527JYBOFS4SXLABDWXQCQQWGXS4UTHDVEQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627100095 (586h8m50.49818313s) [noise=true , address=77.90.31.15:4700 , piece_id=YHWM4LC7OT57QUKZGNBUEYKHZBMSVWFXU7ZL3L462QVDFBCMRTJQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627184977 (586h8m50.498268022s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPEBE2BBZO2EH53A5OY5MINJUKJSCN7BEN2OM5MBMPO7M7MRXKPQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627396790 (586h8m50.498479825s) [noise=true , address=145.79.227.47:4265 , piece_id=BXBESI5LTWBSFLBKTHW5RC4PIDHHHA32THEQ6KS6JIHS3BJTQ5DA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627704056 (586h8m50.498787091s) [noise=true , address=145.79.227.45:4230 , piece_id=JQJIWPAQQUS4YTYH4CRN6TOUSIDNUWSAUZGOMRETBQDUTM7QW65Q, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627759901 (586h8m50.498842936s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628031499 (586h8m50.499114534s) [noise=true , address=77.90.31.21:4880 , piece_id=W4BACYY6ERKB6JDDH2G4CTRK5BGYLR36WM22SY5UD6XY57H2X5EQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628221511 (586h8m50.499304556s) [noise=true , address=145.79.227.20:4439 , piece_id=FCBPUA2VG6Q6THQ6RK36WUJBVGBV6QXYLZH5BWIRQV6DVOG6KFHA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628375484 (586h8m50.499458519s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.629076749 (586h8m50.500159804s) [noise=true , address=77.90.31.12:4590 , piece_id=GZH45VSG3Y6JGMRBJKGT4ZQVHODT72WEMEADYF4NM5FEWGYZMKPA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.629604844 (586h8m50.500687889s) [noise=true , address=145.79.227.15:4875 , piece_id=JYAIPK357MEZYKPE5ZR2BD27PIDPWM5XSMJEXP36DQFS3KVAMZMQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.630977147 (586h8m50.502060182s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:52.326940740 (586h8m52.198023775s) [upload=513837]: Next returned nil segment Dec 30 01:37:52.326953133 (586h8m52.198036168s) [upload=513837]: Waiting for error group managing segments... Dec 30 01:37:52.599100139 (586h8m52.470183184s) [noise=true , address=145.79.227.13:4802 , piece_id=66IYTXYJJQOASFN55SNCRKQ6XTEG7RX3U6KZKL6BYZ3LGSG5SFEA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.607723930 (586h8m52.478806985s) [noise=true , address=145.79.227.45:4230 , piece_id=JQJIWPAQQUS4YTYH4CRN6TOUSIDNUWSAUZGOMRETBQDUTM7QW65Q, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.608660193 (586h8m52.479743238s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPEBE2BBZO2EH53A5OY5MINJUKJSCN7BEN2OM5MBMPO7M7MRXKPQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.609548224 (586h8m52.480631269s) [noise=true , address=145.79.227.46:4603 , piece_id=BY7QVLNIAIIJYXEZKOIE2HK53WVX5UZAKRNSBBIGUNMMRPXWHY2Q, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.611079500 (586h8m52.482162545s) [noise=true , address=145.79.227.15:4875 , piece_id=JYAIPK357MEZYKPE5ZR2BD27PIDPWM5XSMJEXP36DQFS3KVAMZMQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.617429281 (586h8m52.488512326s) [noise=true , address=145.79.227.38:4977 , piece_id=T2HZLMIQZE5GS6ZJ6V527JYBOFS4SXLABDWXQCQQWGXS4UTHDVEQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.629767504 (586h8m52.500850539s) [noise=true , address=145.79.227.19:4079 , piece_id=5EAL2NGIHFXMYBCBHYJHX5CQC6LCALTAXH5ZVHIKAWSX7BEPLJMQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.644911611 (586h8m52.515994646s) [noise=true , address=145.79.227.20:4439 , piece_id=FCBPUA2VG6Q6THQ6RK36WUJBVGBV6QXYLZH5BWIRQV6DVOG6KFHA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.704235833 (586h8m52.575318878s) [noise=true , address=145.79.227.47:4265 , piece_id=BXBESI5LTWBSFLBKTHW5RC4PIDHHHA32THEQ6KS6JIHS3BJTQ5DA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.798030951 (586h8m52.669113996s) [noise=true , address=77.90.31.12:4590 , piece_id=GZH45VSG3Y6JGMRBJKGT4ZQVHODT72WEMEADYF4NM5FEWGYZMKPA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.813278014 (586h8m52.684361059s) [noise=true , address=77.90.31.21:4880 , piece_id=W4BACYY6ERKB6JDDH2G4CTRK5BGYLR36WM22SY5UD6XY57H2X5EQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.991422325 (586h8m52.86250537s) [noise=true , address=45.91.116.7:4012 , piece_id=X4HFHRLPY5N3YESK3FDUV3L7L5ZSAX7HPB34FLZ2VPKJVGPKKOTQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:53.537779026 (586h8m53.408862061s) [noise=true , address=77.90.31.15:4700 , piece_id=YHWM4LC7OT57QUKZGNBUEYKHZBMSVWFXU7ZL3L462QVDFBCMRTJQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:53.537971512 (586h8m53.409054547s) [seg_pos={0 0}, upload=513837]: Segment reached optimal threshold of 13 pieces. Dec 30 01:37:53.538267857 (586h8m53.409350902s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection Dec 30 01:37:53.538277455 (586h8m53.40936049s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538569520 (586h8m53.409652555s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 01:37:53.538578448 (586h8m53.409661483s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538793167 (586h8m53.409876212s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 01:37:53.538801633 (586h8m53.409884678s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538822843 (586h8m53.409905889s) [seg_pos={0 0}, upload=513837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:37:53.538903917 (586h8m53.409986963s) [upload=513837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:37:53.538909979 (586h8m53.409993024s) [upload=513837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:37:53.833385321 (586h8m53.704468367s) [upload=513837]: Done uploading segments... Dec 30 01:37:53.833414106 (586h8m53.704497141s) [upload=513837]: Upload finished. err: <nil> Duration: 6.020301567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X...X....X.X.....X.................XX.....X.X......X.X..XX] duration: 30.00387438s trace: https://jaeger.dev.storj.io/trace/1b6d44ecfce121e7 FAIL (2025-12-30 01:38:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:37:50.379992278 (586h8m50.251075323s) [upload=513837]: Starting upload Dec 30 01:37:50.380101345 (586h8m50.25118439s) [upload=513837]: Done starting upload Dec 30 01:37:50.380205363 (586h8m50.251288398s) [upload=513837]: Uploading segments... Dec 30 01:37:50.380373043 (586h8m50.251456078s) [upload=513837]: Got next segment. Inline: false Dec 30 01:37:50.380414551 (586h8m50.251497587s) [upload=513837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:37:50.380420613 (586h8m50.251503658s) [upload=513837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:37:50.625426289 (586h8m50.496509324s) [seg_pos={0 0}, upload=513837]: Begin upload segment... Dec 30 01:37:50.625474971 (586h8m50.496558006s) [seg_pos={0 0}, upload=513837]: Done begin upload segment. Dec 30 01:37:50.625499919 (586h8m50.496582954s) [noise=true , address=145.79.227.13:4802 , piece_id=66IYTXYJJQOASFN55SNCRKQ6XTEG7RX3U6KZKL6BYZ3LGSG5SFEA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.625581664 (586h8m50.496664699s) [noise=true , address=45.91.116.7:4012 , piece_id=X4HFHRLPY5N3YESK3FDUV3L7L5ZSAX7HPB34FLZ2VPKJVGPKKOTQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626419940 (586h8m50.497502985s) [noise=true , address=145.79.227.46:4603 , piece_id=BY7QVLNIAIIJYXEZKOIE2HK53WVX5UZAKRNSBBIGUNMMRPXWHY2Q, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626543516 (586h8m50.497626561s) [noise=true , address=145.79.227.19:4079 , piece_id=5EAL2NGIHFXMYBCBHYJHX5CQC6LCALTAXH5ZVHIKAWSX7BEPLJMQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626725733 (586h8m50.497808768s) [noise=true , address=145.79.227.38:4977 , piece_id=T2HZLMIQZE5GS6ZJ6V527JYBOFS4SXLABDWXQCQQWGXS4UTHDVEQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627100095 (586h8m50.49818313s) [noise=true , address=77.90.31.15:4700 , piece_id=YHWM4LC7OT57QUKZGNBUEYKHZBMSVWFXU7ZL3L462QVDFBCMRTJQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627184977 (586h8m50.498268022s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPEBE2BBZO2EH53A5OY5MINJUKJSCN7BEN2OM5MBMPO7M7MRXKPQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627396790 (586h8m50.498479825s) [noise=true , address=145.79.227.47:4265 , piece_id=BXBESI5LTWBSFLBKTHW5RC4PIDHHHA32THEQ6KS6JIHS3BJTQ5DA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627704056 (586h8m50.498787091s) [noise=true , address=145.79.227.45:4230 , piece_id=JQJIWPAQQUS4YTYH4CRN6TOUSIDNUWSAUZGOMRETBQDUTM7QW65Q, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627759901 (586h8m50.498842936s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628031499 (586h8m50.499114534s) [noise=true , address=77.90.31.21:4880 , piece_id=W4BACYY6ERKB6JDDH2G4CTRK5BGYLR36WM22SY5UD6XY57H2X5EQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628221511 (586h8m50.499304556s) [noise=true , address=145.79.227.20:4439 , piece_id=FCBPUA2VG6Q6THQ6RK36WUJBVGBV6QXYLZH5BWIRQV6DVOG6KFHA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628375484 (586h8m50.499458519s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.629076749 (586h8m50.500159804s) [noise=true , address=77.90.31.12:4590 , piece_id=GZH45VSG3Y6JGMRBJKGT4ZQVHODT72WEMEADYF4NM5FEWGYZMKPA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.629604844 (586h8m50.500687889s) [noise=true , address=145.79.227.15:4875 , piece_id=JYAIPK357MEZYKPE5ZR2BD27PIDPWM5XSMJEXP36DQFS3KVAMZMQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.630977147 (586h8m50.502060182s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:52.326940740 (586h8m52.198023775s) [upload=513837]: Next returned nil segment Dec 30 01:37:52.326953133 (586h8m52.198036168s) [upload=513837]: Waiting for error group managing segments... Dec 30 01:37:52.599100139 (586h8m52.470183184s) [noise=true , address=145.79.227.13:4802 , piece_id=66IYTXYJJQOASFN55SNCRKQ6XTEG7RX3U6KZKL6BYZ3LGSG5SFEA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.607723930 (586h8m52.478806985s) [noise=true , address=145.79.227.45:4230 , piece_id=JQJIWPAQQUS4YTYH4CRN6TOUSIDNUWSAUZGOMRETBQDUTM7QW65Q, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.608660193 (586h8m52.479743238s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPEBE2BBZO2EH53A5OY5MINJUKJSCN7BEN2OM5MBMPO7M7MRXKPQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.609548224 (586h8m52.480631269s) [noise=true , address=145.79.227.46:4603 , piece_id=BY7QVLNIAIIJYXEZKOIE2HK53WVX5UZAKRNSBBIGUNMMRPXWHY2Q, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.611079500 (586h8m52.482162545s) [noise=true , address=145.79.227.15:4875 , piece_id=JYAIPK357MEZYKPE5ZR2BD27PIDPWM5XSMJEXP36DQFS3KVAMZMQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.617429281 (586h8m52.488512326s) [noise=true , address=145.79.227.38:4977 , piece_id=T2HZLMIQZE5GS6ZJ6V527JYBOFS4SXLABDWXQCQQWGXS4UTHDVEQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.629767504 (586h8m52.500850539s) [noise=true , address=145.79.227.19:4079 , piece_id=5EAL2NGIHFXMYBCBHYJHX5CQC6LCALTAXH5ZVHIKAWSX7BEPLJMQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.644911611 (586h8m52.515994646s) [noise=true , address=145.79.227.20:4439 , piece_id=FCBPUA2VG6Q6THQ6RK36WUJBVGBV6QXYLZH5BWIRQV6DVOG6KFHA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.704235833 (586h8m52.575318878s) [noise=true , address=145.79.227.47:4265 , piece_id=BXBESI5LTWBSFLBKTHW5RC4PIDHHHA32THEQ6KS6JIHS3BJTQ5DA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.798030951 (586h8m52.669113996s) [noise=true , address=77.90.31.12:4590 , piece_id=GZH45VSG3Y6JGMRBJKGT4ZQVHODT72WEMEADYF4NM5FEWGYZMKPA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.813278014 (586h8m52.684361059s) [noise=true , address=77.90.31.21:4880 , piece_id=W4BACYY6ERKB6JDDH2G4CTRK5BGYLR36WM22SY5UD6XY57H2X5EQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.991422325 (586h8m52.86250537s) [noise=true , address=45.91.116.7:4012 , piece_id=X4HFHRLPY5N3YESK3FDUV3L7L5ZSAX7HPB34FLZ2VPKJVGPKKOTQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:53.537779026 (586h8m53.408862061s) [noise=true , address=77.90.31.15:4700 , piece_id=YHWM4LC7OT57QUKZGNBUEYKHZBMSVWFXU7ZL3L462QVDFBCMRTJQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:53.537971512 (586h8m53.409054547s) [seg_pos={0 0}, upload=513837]: Segment reached optimal threshold of 13 pieces. Dec 30 01:37:53.538267857 (586h8m53.409350902s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection Dec 30 01:37:53.538277455 (586h8m53.40936049s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538569520 (586h8m53.409652555s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 01:37:53.538578448 (586h8m53.409661483s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538793167 (586h8m53.409876212s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 01:37:53.538801633 (586h8m53.409884678s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538822843 (586h8m53.409905889s) [seg_pos={0 0}, upload=513837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:37:53.538903917 (586h8m53.409986963s) [upload=513837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:37:53.538909979 (586h8m53.409993024s) [upload=513837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:37:53.833385321 (586h8m53.704468367s) [upload=513837]: Done uploading segments... Dec 30 01:37:53.833414106 (586h8m53.704497141s) [upload=513837]: Upload finished. err: <nil> Duration: 30.003897834s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)