Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.234916874s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:33:01 UTC) orbiter warning: context deadline exceeded Duration: 6.234960716s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.855234346s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:32:09 UTC) orbiter warning: context deadline exceeded Duration: 5.855262108s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X..............................X.X.X.X.XX....X.X.X...] duration: 6.229973993s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:32:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 6.230001465s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.893664463s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:31:45 UTC) orbiter warning: context deadline exceeded Duration: 5.893685783s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.259567751s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:31:11 UTC) orbiter warning: context deadline exceeded Duration: 6.259597899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.......X..............................X.X.X.X.XX....X.X.X..] duration: 6.00225409s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:31:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 6.002281382s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.832494214s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:30:35 UTC) orbiter warning: context deadline exceeded Duration: 5.832532227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.......X..............................X.X.X.X.XX....X.X.X.] duration: 8.054581582s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:30:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 8.054611598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.......X..............................X.X.X.X.XX....X.X.X] duration: 30.002389499s trace: https://jaeger.dev.storj.io/trace/e50feccdf855085 FAIL (2025-12-26 23:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q) context deadline exceeded ======LOGS====== Dec 26 23:29:22.756810425 (512h0m22.62789347s) [upload=449270]: Starting upload Dec 26 23:29:22.756888433 (512h0m22.627971478s) [upload=449270]: Done starting upload Dec 26 23:29:22.757038078 (512h0m22.628121133s) [upload=449270]: Uploading segments... Dec 26 23:29:22.757162165 (512h0m22.6282452s) [upload=449270]: Got next segment. Inline: false Dec 26 23:29:22.757203654 (512h0m22.628286689s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:29:22.757208162 (512h0m22.628291197s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:29:23.021104849 (512h0m22.892187894s) [seg_pos={0 0}, upload=449270]: Begin upload segment... Dec 26 23:29:23.021165694 (512h0m22.892248729s) [seg_pos={0 0}, upload=449270]: Done begin upload segment. Dec 26 23:29:23.021280483 (512h0m22.892363528s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021288709 (512h0m22.892371754s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.021456448 (512h0m22.892539493s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.022245640 (512h0m22.893328675s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.023406922 (512h0m22.894489967s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.030667898 (512h0m22.901751053s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.031984896 (512h0m22.903067931s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033698288 (512h0m22.904781333s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.033942994 (512h0m22.905026039s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.035690882 (512h0m22.906773917s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.036655409 (512h0m22.907738464s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.037874871 (512h0m22.908957916s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.039624583 (512h0m22.910707628s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.040256897 (512h0m22.911339942s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042088945 (512h0m22.913171991s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:23.042879702 (512h0m22.913962747s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Uploading piece... Dec 26 23:29:24.739869576 (512h0m24.610952611s) [upload=449270]: Next returned nil segment Dec 26 23:29:24.739884054 (512h0m24.610967089s) [upload=449270]: Waiting for error group managing segments... Dec 26 23:29:25.023209585 (512h0m24.89429262s) [noise=true , address=145.79.227.17:4971 , piece_id=BK4WUTTOHEAGLJAI3TAZXY2RMVH56SYZGS5A4JJFNHWOLTDNNSAQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.038816673 (512h0m24.909899708s) [noise=true , address=145.79.227.42:4121 , piece_id=NU2JHDFXFN6SY662O3YUKYA6OYRHLRSM7MMLQG4446JSXBHBTNUA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.040020275 (512h0m24.91110331s) [noise=true , address=145.79.227.48:4302 , piece_id=EVM2UZBVLKX35425YPJE6MOJ4OHP26QI5XYHBMVFG5DH4SZY6NGA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.046609082 (512h0m24.917692127s) [noise=true , address=145.79.227.11:4732 , piece_id=OYX5DPQDFSVTWMLSJIFV2TLL4JQKAGVPIWALMNSWIR6FJXPYNDPQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.049410005 (512h0m24.92049305s) [noise=true , address=145.79.227.47:4266 , piece_id=6QK5BUG3S2QCJVVQ2DMUSAFP5FNY3OFYTPX2J3B5LTAMO5AR4TIQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.051015772 (512h0m24.922098807s) [noise=true , address=145.79.227.45:4229 , piece_id=4K72QWKFXLNHIMLSANKGSWBO35LC5GAN36QLPJF55MGVLBXU3DHA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.056805787 (512h0m24.927888832s) [noise=true , address=145.79.227.37:4567 , piece_id=G42YXXCAOR3TJIPNAHYOHH2QA6FALPOOUJQK3FRKDWZVS4ZD3ROQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.236396903 (512h0m25.107479948s) [noise=true , address=145.79.227.13:4804 , piece_id=A3CXVEHFRXI54T6ATKEZYUNPPIJ6KWENCPEJZG6VB5GGSGGF436A, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.258471718 (512h0m25.129554753s) [noise=true , address=77.90.31.10:4519 , piece_id=7F3TQ4XFSIYOQ5O4KN6RKXL5GF5DDWPCYYZHRHXRTXAVNWGSOFSA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.281602180 (512h0m25.152685216s) [noise=true , address=45.91.116.6:4975 , piece_id=7DFAXR3RJH3CC7YGCZS2Z4AV3Y4OVU43F3FESGEXOUU7W2KJ7EVA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.289377055 (512h0m25.16046009s) [noise=true , address=45.91.116.7:4009 , piece_id=C6HPB5QYPD4C3X6HOKATODOK24QLVD4NDTJDKNSJEH5IJPBVZABQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.321506794 (512h0m25.192589829s) [noise=true , address=45.91.116.5:4300 , piece_id=WX5WPYNNCVTPJ54FV2AKTPCWCZ5WJEMO6DWL2IDEZUJSVH7NAR4Q, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.337992606 (512h0m25.209075641s) [noise=true , address=77.90.31.13:4629 , piece_id=I3O6UXXPHY42WGD5BCTP5URSW4UKFZLNFTNCSNAIIDVDPYGFZTJA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: <nil> Dec 26 23:29:25.338010801 (512h0m25.209093836s) [seg_pos={0 0}, upload=449270]: Segment reached optimal threshold of 13 pieces. Dec 26 23:29:25.338188900 (512h0m25.209271935s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338197596 (512h0m25.209280631s) [noise=true , address=77.90.31.17:4770 , piece_id=VMXZUK3WAFVSDHLIVNJAXD63H4NCHTLN523YSWHTM27YD3FC2APQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338525982 (512h0m25.209609027s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48708->45.91.116.10:4081: use of closed network connection Dec 26 23:29:25.338534438 (512h0m25.209617483s) [noise=true , address=45.91.116.10:4081 , piece_id=6AACECJXQSBEHB3EGR4MZPGICOPYVCLFFJOJHEGCDSG4H3XUDRTA, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338669816 (512h0m25.209752851s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 26 23:29:25.338706636 (512h0m25.209789671s) [noise=true , address=77.90.31.18:4805 , piece_id=NOFX5HLOVOL5PMVAQLMLRR7M5BJKC7GMOTZFA7WCZAA5HY2POBVQ, seg_pos={0 0}, upload=449270]: Long tail cancellation due to optimal threshold Dec 26 23:29:25.338726373 (512h0m25.209809418s) [seg_pos={0 0}, upload=449270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:29:25.338798391 (512h0m25.209881426s) [upload=449270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:29:25.338803480 (512h0m25.209886515s) [upload=449270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:29:25.633003348 (512h0m25.504086393s) [upload=449270]: Done uploading segments... Dec 26 23:29:25.633108388 (512h0m25.504191433s) [upload=449270]: Upload finished. err: <nil> Duration: 30.002410478s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.217842924s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:29:27 UTC) orbiter warning: context deadline exceeded Duration: 6.217870805s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.......X..............................X.X.X.X.XX....X.X.] duration: 16.343954836s trace: https://jaeger.dev.storj.io/trace/fa7a2ac26cbf8ad FAIL (2025-12-26 23:28:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A) context deadline exceeded ======LOGS====== Dec 26 23:27:51.598399489 (511h58m51.469482524s) [upload=449251]: Starting upload Dec 26 23:27:51.598502145 (511h58m51.46958521s) [upload=449251]: Done starting upload Dec 26 23:27:51.598595983 (511h58m51.469679038s) [upload=449251]: Uploading segments... Dec 26 23:27:51.598696234 (511h58m51.469779279s) [upload=449251]: Got next segment. Inline: false Dec 26 23:27:51.598737111 (511h58m51.469820157s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:27:51.598743323 (511h58m51.469826378s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:27:51.841567824 (511h58m51.712650869s) [seg_pos={0 0}, upload=449251]: Begin upload segment... Dec 26 23:27:51.841616236 (511h58m51.712699271s) [seg_pos={0 0}, upload=449251]: Done begin upload segment. Dec 26 23:27:51.841641194 (511h58m51.712724229s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.841671933 (511h58m51.712754968s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842203494 (511h58m51.71328653s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842584951 (511h58m51.713667996s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842840809 (511h58m51.713923844s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843251670 (511h58m51.714334715s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843530411 (511h58m51.714613446s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843857985 (511h58m51.71494102s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844225374 (511h58m51.715308409s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844596762 (511h58m51.715679807s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.846561803 (511h58m51.717644838s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.847861148 (511h58m51.718944183s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.849012220 (511h58m51.720095255s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.850316984 (511h58m51.721400029s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.851405207 (511h58m51.722488242s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.852456639 (511h58m51.723539684s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:53.556959120 (511h58m53.428042165s) [upload=449251]: Next returned nil segment Dec 26 23:27:53.556989167 (511h58m53.428072202s) [upload=449251]: Waiting for error group managing segments... Dec 26 23:27:53.844783598 (511h58m53.715866643s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.851886403 (511h58m53.722969448s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.860937488 (511h58m53.732020533s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867202107 (511h58m53.738285162s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867515253 (511h58m53.738598288s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.869419589 (511h58m53.740502624s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.074603758 (511h58m53.945686794s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.168599989 (511h58m54.039683024s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.188116953 (511h58m54.059199988s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.475777599 (511h58m54.346860634s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.996276903 (511h58m54.867359938s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.243663372 (511h58m55.114746407s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486969831 (511h58m55.358052876s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486998786 (511h58m55.358081831s) [seg_pos={0 0}, upload=449251]: Segment reached optimal threshold of 13 pieces. Dec 26 23:27:55.487329226 (511h58m55.358412271s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487340747 (511h58m55.358423802s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487395672 (511h58m55.358478717s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487407885 (511h58m55.35849093s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487564944 (511h58m55.358647989s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487572880 (511h58m55.358655925s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487593368 (511h58m55.358676403s) [seg_pos={0 0}, upload=449251]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:27:55.487685253 (511h58m55.358768298s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:27:55.487691866 (511h58m55.358774911s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:27:55.779554575 (511h58m55.65063762s) [upload=449251]: Done uploading segments... Dec 26 23:27:55.779577028 (511h58m55.650660063s) [upload=449251]: Upload finished. err: <nil> Duration: 16.343970837s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.896815579s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:28:33 UTC) orbiter warning: context deadline exceeded Duration: 5.896841789s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X.......X..............................X.X.X.X.XX....X.X] duration: 30.003032179s trace: https://jaeger.dev.storj.io/trace/fa7a2ac26cbf8ad FAIL (2025-12-26 23:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A) context deadline exceeded ======LOGS====== Dec 26 23:27:51.598399489 (511h58m51.469482524s) [upload=449251]: Starting upload Dec 26 23:27:51.598502145 (511h58m51.46958521s) [upload=449251]: Done starting upload Dec 26 23:27:51.598595983 (511h58m51.469679038s) [upload=449251]: Uploading segments... Dec 26 23:27:51.598696234 (511h58m51.469779279s) [upload=449251]: Got next segment. Inline: false Dec 26 23:27:51.598737111 (511h58m51.469820157s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:27:51.598743323 (511h58m51.469826378s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:27:51.841567824 (511h58m51.712650869s) [seg_pos={0 0}, upload=449251]: Begin upload segment... Dec 26 23:27:51.841616236 (511h58m51.712699271s) [seg_pos={0 0}, upload=449251]: Done begin upload segment. Dec 26 23:27:51.841641194 (511h58m51.712724229s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.841671933 (511h58m51.712754968s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842203494 (511h58m51.71328653s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842584951 (511h58m51.713667996s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.842840809 (511h58m51.713923844s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843251670 (511h58m51.714334715s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843530411 (511h58m51.714613446s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.843857985 (511h58m51.71494102s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844225374 (511h58m51.715308409s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.844596762 (511h58m51.715679807s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.846561803 (511h58m51.717644838s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.847861148 (511h58m51.718944183s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.849012220 (511h58m51.720095255s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.850316984 (511h58m51.721400029s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.851405207 (511h58m51.722488242s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:51.852456639 (511h58m51.723539684s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Uploading piece... Dec 26 23:27:53.556959120 (511h58m53.428042165s) [upload=449251]: Next returned nil segment Dec 26 23:27:53.556989167 (511h58m53.428072202s) [upload=449251]: Waiting for error group managing segments... Dec 26 23:27:53.844783598 (511h58m53.715866643s) [noise=true , address=145.79.227.19:4079 , piece_id=Z5DNPFZFEODNICTDJX7B4SC3EJXR6HF5LA4SJRSCXHF6U6UEFS7A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.851886403 (511h58m53.722969448s) [noise=true , address=145.79.227.36:4517 , piece_id=MYHNRWBYUWGF4XHNGP6MNSR2EX4NKVDK6SERBNER6UEU7Z6RINXA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.860937488 (511h58m53.732020533s) [noise=true , address=145.79.227.15:4874 , piece_id=QAIG5MOKVWRA7QTYSLDGKSYICFEW3RTCGLPYCPTPLH2W7MFRWI6A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867202107 (511h58m53.738285162s) [noise=true , address=145.79.227.12:4767 , piece_id=6IE2GEK3JYRN7KNZUKA5GSVLU2ACHXT75B6B3NAKOF26NJEBM4CQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.867515253 (511h58m53.738598288s) [noise=true , address=145.79.227.47:4267 , piece_id=RQLVEP7QJWEFNAHKMHMMPEAWSDH53CZFDSKGBLEFFTDKLOSNGRPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:53.869419589 (511h58m53.740502624s) [noise=true , address=145.79.227.13:4804 , piece_id=7KOYT64CJDKPR4LGYCN75LDVTTNHNLFAK7JAZI5ETIVLCHKNDPPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.074603758 (511h58m53.945686794s) [noise=true , address=45.91.116.12:4228 , piece_id=U6ZDQVEZQIXKSJWUEROZN7RXAOA3YQ5WK6XEG3YJ6W7DMAX3N57A, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.168599989 (511h58m54.039683024s) [noise=true , address=45.91.116.8:4046 , piece_id=CAHDSCAWPAYHQYUQCI6UZ6FNLQAVT35NSYTVFTCJSQJAOFCCI6TA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.188116953 (511h58m54.059199988s) [noise=true , address=77.90.31.19:4842 , piece_id=5J7XL7STKUPXR4UTUGYN24RBQUAQW4QWN6YIKSIIP433VK6PMMYA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.475777599 (511h58m54.346860634s) [noise=true , address=77.90.31.11:4553 , piece_id=YK6VVJJ2I6Q7XX7WTMF4DGELD5RGORG3EPUMA7E5BYY7Z3JOXLXQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:54.996276903 (511h58m54.867359938s) [noise=true , address=45.91.116.11:4118 , piece_id=XHE6VOUSZATVMDQKIOXAHL4CLFOFHVXGNN4N3SA6NXKECYZHF5ZA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.243663372 (511h58m55.114746407s) [noise=true , address=145.79.227.44:4193 , piece_id=I6PZX6YNSSJQ6AIC5TX36FVH7C7TJU6LVDIS5QKFZ2UI3ZF4DAKQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486969831 (511h58m55.358052876s) [noise=true , address=77.90.31.15:4697 , piece_id=5AHWFOFJYT3ZT55KTI6VDM4YUPUAFEI5UZRC7XSKZ726AXZ52X3Q, seg_pos={0 0}, upload=449251]: Done uploading piece. err: <nil> Dec 26 23:27:55.486998786 (511h58m55.358081831s) [seg_pos={0 0}, upload=449251]: Segment reached optimal threshold of 13 pieces. Dec 26 23:27:55.487329226 (511h58m55.358412271s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487340747 (511h58m55.358423802s) [noise=true , address=77.90.31.14:4662 , piece_id=JTIBSX3EN3VDC6E6ETPAVB23TKEJ3W6WA7J7XPIQMZFGURHEGYPQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487395672 (511h58m55.358478717s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487407885 (511h58m55.35849093s) [noise=true , address=45.91.116.5:4299 , piece_id=A2ZUXHJ3Z3LQHLFR5DO6MA6JEO3XCJWOEBCNQCRTRH2XMO5ECKTQ, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487564944 (511h58m55.358647989s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 23:27:55.487572880 (511h58m55.358655925s) [noise=true , address=145.79.227.34:4907 , piece_id=GLU457XO3XARZXT7AM4GXG525I3MR7UUHVZBO5GLALOLDQNAS2EA, seg_pos={0 0}, upload=449251]: Long tail cancellation due to optimal threshold Dec 26 23:27:55.487593368 (511h58m55.358676403s) [seg_pos={0 0}, upload=449251]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:27:55.487685253 (511h58m55.358768298s) [upload=449251]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:27:55.487691866 (511h58m55.358774911s) [upload=449251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:27:55.779554575 (511h58m55.65063762s) [upload=449251]: Done uploading segments... Dec 26 23:27:55.779577028 (511h58m55.650660063s) [upload=449251]: Upload finished. err: <nil> Duration: 30.003056375s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.......X..............................X.X.X.X.XX....X.] duration: 6.01916603s trace: https://jaeger.dev.storj.io/trace/78807657ce9ceff1 FAIL (2025-12-26 23:27:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.505609383 (511h57m53.376692429s) [upload=449236]: Starting upload Dec 26 23:26:53.505662284 (511h57m53.376745319s) [upload=449236]: Done starting upload Dec 26 23:26:53.505715856 (511h57m53.376798891s) [upload=449236]: Uploading segments... Dec 26 23:26:53.505853358 (511h57m53.376936393s) [upload=449236]: Got next segment. Inline: false Dec 26 23:26:53.505895448 (511h57m53.376978483s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:26:53.505899556 (511h57m53.376982591s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:26:53.752751397 (511h57m53.623834432s) [seg_pos={0 0}, upload=449236]: Begin upload segment... Dec 26 23:26:53.752814487 (511h57m53.623897522s) [seg_pos={0 0}, upload=449236]: Done begin upload segment. Dec 26 23:26:53.752856136 (511h57m53.623939171s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.752880012 (511h57m53.623963047s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753334968 (511h57m53.624418013s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753677371 (511h57m53.624760406s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754145051 (511h57m53.625228086s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754551044 (511h57m53.625634089s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754867196 (511h57m53.625950231s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.755164061 (511h57m53.626247096s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.756746094 (511h57m53.627829139s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.757479240 (511h57m53.628562275s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.759848601 (511h57m53.630931636s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.762664954 (511h57m53.633747989s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.764549372 (511h57m53.635632407s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.766050160 (511h57m53.637133195s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.769742941 (511h57m53.640825976s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.771062714 (511h57m53.642145749s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:55.430505414 (511h57m55.301588449s) [upload=449236]: Next returned nil segment Dec 26 23:26:55.430525082 (511h57m55.301608117s) [upload=449236]: Waiting for error group managing segments... Dec 26 23:26:55.751239376 (511h57m55.622322421s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.756350789 (511h57m55.627433824s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759738971 (511h57m55.630822006s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759770721 (511h57m55.630853756s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.761504331 (511h57m55.632587377s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.762883748 (511h57m55.633966794s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.773095964 (511h57m55.644178999s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.777619698 (511h57m55.648702733s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.793659660 (511h57m55.664742695s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.977319374 (511h57m55.848402409s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.013973318 (511h57m55.885056353s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.075286767 (511h57m55.946369813s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081874231 (511h57m55.952957276s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081891103 (511h57m55.952974148s) [seg_pos={0 0}, upload=449236]: Segment reached optimal threshold of 13 pieces. Dec 26 23:26:56.082082457 (511h57m55.953165502s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082089861 (511h57m55.953172906s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082388340 (511h57m55.953471375s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection Dec 26 23:26:56.082395604 (511h57m55.953478649s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082479042 (511h57m55.953562097s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082502688 (511h57m55.953585753s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082558364 (511h57m55.953641409s) [seg_pos={0 0}, upload=449236]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:26:56.082677550 (511h57m55.953760586s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:26:56.082685796 (511h57m55.953768841s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:26:56.368370360 (511h57m56.239453395s) [upload=449236]: Done uploading segments... Dec 26 23:26:56.368391740 (511h57m56.239474785s) [upload=449236]: Upload finished. err: <nil> Duration: 6.019180027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.......X..............................X.X.X.X.XX....X] duration: 30.002523404s trace: https://jaeger.dev.storj.io/trace/78807657ce9ceff1 FAIL (2025-12-26 23:27:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.505609383 (511h57m53.376692429s) [upload=449236]: Starting upload Dec 26 23:26:53.505662284 (511h57m53.376745319s) [upload=449236]: Done starting upload Dec 26 23:26:53.505715856 (511h57m53.376798891s) [upload=449236]: Uploading segments... Dec 26 23:26:53.505853358 (511h57m53.376936393s) [upload=449236]: Got next segment. Inline: false Dec 26 23:26:53.505895448 (511h57m53.376978483s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:26:53.505899556 (511h57m53.376982591s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:26:53.752751397 (511h57m53.623834432s) [seg_pos={0 0}, upload=449236]: Begin upload segment... Dec 26 23:26:53.752814487 (511h57m53.623897522s) [seg_pos={0 0}, upload=449236]: Done begin upload segment. Dec 26 23:26:53.752856136 (511h57m53.623939171s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.752880012 (511h57m53.623963047s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753334968 (511h57m53.624418013s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.753677371 (511h57m53.624760406s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754145051 (511h57m53.625228086s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754551044 (511h57m53.625634089s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.754867196 (511h57m53.625950231s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.755164061 (511h57m53.626247096s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.756746094 (511h57m53.627829139s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.757479240 (511h57m53.628562275s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.759848601 (511h57m53.630931636s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.762664954 (511h57m53.633747989s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.764549372 (511h57m53.635632407s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.766050160 (511h57m53.637133195s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.769742941 (511h57m53.640825976s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:53.771062714 (511h57m53.642145749s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Uploading piece... Dec 26 23:26:55.430505414 (511h57m55.301588449s) [upload=449236]: Next returned nil segment Dec 26 23:26:55.430525082 (511h57m55.301608117s) [upload=449236]: Waiting for error group managing segments... Dec 26 23:26:55.751239376 (511h57m55.622322421s) [noise=true , address=145.79.227.37:4566 , piece_id=ZGQ64Q3P6C4EYHEO2TV2LNIRFTOWWXMDHNCJUAE3GNVNWP2GZJMQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.756350789 (511h57m55.627433824s) [noise=true , address=145.79.227.45:4231 , piece_id=BF65NXZ4XB2FMRU3FKKCMWACOU3FIMBAB2ECMSEWAQNDDLHA7KEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759738971 (511h57m55.630822006s) [noise=true , address=145.79.227.49:4337 , piece_id=YFHMJF7VEX3JQGLT4BTT2EI7CRVBIXVENUIUCB5KJNC6ANKINXCQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.759770721 (511h57m55.630853756s) [noise=true , address=145.79.227.46:4601 , piece_id=GIZBX6VDLGQEOCS2MXCCZC2G4UQITOVLOL4NTXBWDGI7VPI7M7CA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.761504331 (511h57m55.632587377s) [noise=true , address=145.79.227.48:4301 , piece_id=PHZ3DBRWYZJHT5FQBQJQRGLUUM2K4TQMQUBQ2LV2TQL22XPCPLOA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.762883748 (511h57m55.633966794s) [noise=true , address=145.79.227.23:4545 , piece_id=3AB3XAYT55RI4SPNMBBPPOOOG42MJVMFAEVXHUEI5NHHIC63LCFA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.773095964 (511h57m55.644178999s) [noise=true , address=145.79.227.7:4295 , piece_id=77AWXKRCCCX7LB77SS66PS3YWEUGIKNZQ4MOR24F24EOMQUZKVKQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.777619698 (511h57m55.648702733s) [noise=true , address=145.79.227.11:4729 , piece_id=GJFZCLMQTGIWZGYXJQQ5PDTIHTZBNIF4W6ZXRATAAR5QGT2U4ROA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.793659660 (511h57m55.664742695s) [noise=true , address=145.79.227.51:4409 , piece_id=6DA2ZNTV2UEK54EDUAEOJFFABD7JC4GZDH3HQQHT73CU6CPAJ35Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:55.977319374 (511h57m55.848402409s) [noise=true , address=77.90.31.14:4662 , piece_id=5IERUYB7PVCISTZAVLQACT256HDXTUWOE7KOEY2JXJ2WTJ43ZMEQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.013973318 (511h57m55.885056353s) [noise=true , address=45.91.116.9:4263 , piece_id=OKDFP3FTRPF3LEECF7RPVU4EJT7Z5UYPCVDVPDZ76W7XGAF6YIGQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.075286767 (511h57m55.946369813s) [noise=true , address=45.91.116.5:4298 , piece_id=7NGLZFGZBYZCO55HAD6EEV7D372SWZNLO2VRFQEA75ZIV6ELJEQQ, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081874231 (511h57m55.952957276s) [noise=true , address=45.91.116.8:4046 , piece_id=Q2TJ5FGZUE4ECRL4SELEMRB62UNMQRYELYI7FTCWS66SO32WEOAA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: <nil> Dec 26 23:26:56.081891103 (511h57m55.952974148s) [seg_pos={0 0}, upload=449236]: Segment reached optimal threshold of 13 pieces. Dec 26 23:26:56.082082457 (511h57m55.953165502s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082089861 (511h57m55.953172906s) [noise=true , address=45.91.116.7:4009 , piece_id=TRL7ZNDZFARUJHKD3MEJAT2AMPUPCRUZQUQI7JWBBAZ5YBNQ6KSA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082388340 (511h57m55.953471375s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50766->145.79.227.19:4077: use of closed network connection Dec 26 23:26:56.082395604 (511h57m55.953478649s) [noise=true , address=145.79.227.19:4077 , piece_id=2C5ACPFVFVLTVHOEPQ6OONFHNCTPVFDDFXOFZBNFP5H6LFKP3ASA, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082479042 (511h57m55.953562097s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 23:26:56.082502688 (511h57m55.953585753s) [noise=true , address=77.90.31.13:4629 , piece_id=AJ2ZSIPI7LUTORNEJ7KRWLQUTWWS7MWFFQQ5LWO3PMWU7NC2HP2Q, seg_pos={0 0}, upload=449236]: Long tail cancellation due to optimal threshold Dec 26 23:26:56.082558364 (511h57m55.953641409s) [seg_pos={0 0}, upload=449236]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:26:56.082677550 (511h57m55.953760586s) [upload=449236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:26:56.082685796 (511h57m55.953768841s) [upload=449236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:26:56.368370360 (511h57m56.239453395s) [upload=449236]: Done uploading segments... Dec 26 23:26:56.368391740 (511h57m56.239474785s) [upload=449236]: Upload finished. err: <nil> Duration: 30.00254783s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.886469178s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:27:18 UTC) orbiter warning: context deadline exceeded Duration: 5.886517781s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.......X..............................X.X.X.X.XX....] duration: 8.220331097s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 8.220363228s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.863592649s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:26:08 UTC) orbiter warning: context deadline exceeded Duration: 5.863639629s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.115318089s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:25:16 UTC) orbiter warning: context deadline exceeded Duration: 6.115344057s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.......X.......X..............................X.X.X.X.XX...] duration: 5.955166751s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:25:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 5.955212689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.......X.......X..............................X.X.X.X.XX..] duration: 6.210962483s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 6.210998792s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.921057232s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:24:44 UTC) orbiter warning: context deadline exceeded Duration: 5.921095395s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.845505437s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:24:13 UTC) orbiter warning: context deadline exceeded Duration: 5.84557478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.......X.......X..............................X.X.X.X.XX.] duration: 24.723998936s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 24.724018442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.......X.......X..............................X.X.X.X.XX] duration: 30.002688518s trace: https://jaeger.dev.storj.io/trace/2c24412a610f56eb FAIL (2025-12-26 23:23:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:23:05.603616059 (511h54m5.474699104s) [upload=449185]: Starting upload Dec 26 23:23:05.603696151 (511h54m5.474779186s) [upload=449185]: Done starting upload Dec 26 23:23:05.603788898 (511h54m5.474871933s) [upload=449185]: Uploading segments... Dec 26 23:23:05.603886854 (511h54m5.474969889s) [upload=449185]: Got next segment. Inline: false Dec 26 23:23:05.603916149 (511h54m5.474999194s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:23:05.603920868 (511h54m5.475003903s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:23:05.852647268 (511h54m5.723730313s) [seg_pos={0 0}, upload=449185]: Begin upload segment... Dec 26 23:23:05.852769261 (511h54m5.723852296s) [seg_pos={0 0}, upload=449185]: Done begin upload segment. Dec 26 23:23:05.852799479 (511h54m5.723882514s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.852808355 (511h54m5.72389139s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853320691 (511h54m5.724403726s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.853723938 (511h54m5.724806973s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.854130373 (511h54m5.725213408s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855261356 (511h54m5.726344401s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.855824829 (511h54m5.726907865s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.856282851 (511h54m5.727365887s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857319685 (511h54m5.72840273s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.857848763 (511h54m5.728931798s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858345228 (511h54m5.729428273s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.858800325 (511h54m5.72988336s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.862667519 (511h54m5.733750564s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863090704 (511h54m5.734173749s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863495576 (511h54m5.734578621s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:05.863867354 (511h54m5.734950389s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Uploading piece... Dec 26 23:23:07.570106439 (511h54m7.441189474s) [upload=449185]: Next returned nil segment Dec 26 23:23:07.570121899 (511h54m7.441204934s) [upload=449185]: Waiting for error group managing segments... Dec 26 23:23:07.856693962 (511h54m7.727777007s) [noise=true , address=145.79.227.6:4259 , piece_id=UWU5TUBMKWAO4FRGFXEDWOBGNRFJHBJKODAOI7AL3FXEDAPNX3RQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.860553270 (511h54m7.731636315s) [noise=true , address=145.79.227.41:4085 , piece_id=44EKAL2JHINSVZAJ6HMS3VG3HO37VOLQ2EWGHSENYE3FIBYBX4PA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.874397222 (511h54m7.745480267s) [noise=true , address=145.79.227.36:4519 , piece_id=ZMZJXNCW5TAENS4RQJ3X6TXJ3FIIMVNNTXZ2JCXXENK27SKVJDTQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.889965135 (511h54m7.76104818s) [noise=true , address=145.79.227.45:4231 , piece_id=DS6A5K6YJE4FU356BPSKR56SZFMOXA3KOQ7EX66MDBZ7OPDCYQOQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.890583082 (511h54m7.761666127s) [noise=true , address=145.79.227.17:4972 , piece_id=WI3BLDRVYG6XZRIOUOMT4LNB2GKJ5KYNVRYZTCJIVNY2B4FM7BJA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.892022783 (511h54m7.763105828s) [noise=true , address=145.79.227.50:4374 , piece_id=WJWDSSU5YPHDGZOUJS7V27CDI4NQ26SYQU36YSI2DY6Q3SWPX4LQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.899883451 (511h54m7.770966496s) [noise=true , address=145.79.227.42:4122 , piece_id=2RRYX5RNMQ2I3FSD326YJMKYQMZLH34E66SCGGG2S2EXCUII4O4A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.902312215 (511h54m7.77339526s) [noise=true , address=145.79.227.47:4267 , piece_id=VG62Y2XXXJLI7PQ33VA4LHJUW3CNBMJ2LWH7SO3YWUKEEEF2TNCQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:07.971838392 (511h54m7.842921438s) [noise=true , address=145.79.227.3:4150 , piece_id=IL3DELYXSAD3VNOZQ5WVBFU36JEYBVUIUT6KGT4YJX37JINANLUA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.051557429 (511h54m7.922640474s) [noise=true , address=77.90.31.17:4770 , piece_id=IBP74CPPS35TJ5OI7KYAH6J4ERDFGDIW6JO6JIPT6P6CGOYMB7SQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.057346031 (511h54m7.928429077s) [noise=true , address=77.90.31.14:4662 , piece_id=H7UJNB6UAFRGMSCSN7AGQFIA6AR22JFFYVETS7NH5A5UOBEMLZ6A, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.074732869 (511h54m7.945815904s) [noise=true , address=77.90.31.11:4556 , piece_id=QJK7FACGRTZVATMCET2W2UWHQEFU5JDCJ2YW4S5Y7RPBS4T6NDIQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131808819 (511h54m8.002891864s) [noise=true , address=45.91.116.5:4299 , piece_id=UNBIZ5EPSKFQJBBJEDJ7BFTWNEH4ULEPXEHHO6V3L7DG6OORBXZA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: <nil> Dec 26 23:23:08.131835931 (511h54m8.002918976s) [seg_pos={0 0}, upload=449185]: Segment reached optimal threshold of 13 pieces. Dec 26 23:23:08.132083362 (511h54m8.003166397s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132093501 (511h54m8.003176536s) [noise=true , address=45.91.116.9:4263 , piece_id=JERCJ3ETUH7SDEOEI7PADBQZ2CHZL7NY5O7OLS26R5QNKNY3DSKQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132099032 (511h54m8.003182067s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 23:23:08.132108901 (511h54m8.003191936s) [noise=true , address=145.79.227.40:4050 , piece_id=POWJZBHJPUZORIOYXVWI37XUKICFWF36DN44VUP4VAV5XLQ5RLBQ, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132370238 (511h54m8.003453273s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50054->45.91.116.7:4009: use of closed network connection Dec 26 23:23:08.132379366 (511h54m8.003462411s) [noise=true , address=45.91.116.7:4009 , piece_id=VUW4SW6M5BKOHX67L76QZ72DCMN2H3K5IVAJ6OCKUKSUGWBJPJQA, seg_pos={0 0}, upload=449185]: Long tail cancellation due to optimal threshold Dec 26 23:23:08.132402279 (511h54m8.003485324s) [seg_pos={0 0}, upload=449185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:23:08.132539851 (511h54m8.003622886s) [upload=449185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:23:08.132546865 (511h54m8.0036299s) [upload=449185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:23:08.422637417 (511h54m8.293720462s) [upload=449185]: Done uploading segments... Dec 26 23:23:08.422660792 (511h54m8.293743837s) [upload=449185]: Upload finished. err: <nil> Duration: 30.00271085s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.975049525s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:23:25 UTC) orbiter warning: context deadline exceeded Duration: 5.975075563s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.......X.......X..............................X.X.X.X.X] duration: 30.00218s trace: https://jaeger.dev.storj.io/trace/6c9e75584b5f6bf2 FAIL (2025-12-26 23:22:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 23:22:18.091423839 (511h53m17.962506884s) [upload=449174]: Starting upload Dec 26 23:22:18.091520293 (511h53m17.962603338s) [upload=449174]: Done starting upload Dec 26 23:22:18.091689686 (511h53m17.962772751s) [upload=449174]: Uploading segments... Dec 26 23:22:18.091812088 (511h53m17.962895143s) [upload=449174]: Got next segment. Inline: false Dec 26 23:22:18.091863456 (511h53m17.962946492s) [upload=449174]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:22:18.091869858 (511h53m17.962952894s) [upload=449174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:22:18.335518279 (511h53m18.206601324s) [seg_pos={0 0}, upload=449174]: Begin upload segment... Dec 26 23:22:18.335564988 (511h53m18.206648023s) [seg_pos={0 0}, upload=449174]: Done begin upload segment. Dec 26 23:22:18.335587381 (511h53m18.206670416s) [noise=true , address=145.79.227.47:4266 , piece_id=LFBF4PKKFEWHYQRGXUEHMWTVHS26OKSAZXWF5YAHJLIRFEUBRKOA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.335673204 (511h53m18.20675625s) [noise=true , address=145.79.227.33:4869 , piece_id=XMFA6KKJV35XYM2JTRBKQBQLHVTTHFCNRC2ZAESBT5FRFES64NRA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.336109385 (511h53m18.20719243s) [noise=true , address=145.79.227.37:4566 , piece_id=CNSCGTMCFEHWELZONKUNC2TEJFNIAH6TC4XWUEZB7A5YW2TTVWOQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.336505929 (511h53m18.207588964s) [noise=true , address=145.79.227.29:4943 , piece_id=25IQXVLTMAOLS3K2VAZ4SL5ECLZ3BPNJTXOPDNM47QTXHX3M5HVA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.336943723 (511h53m18.208026758s) [noise=true , address=45.91.116.7:4010 , piece_id=ILPUYVGX4T6UEGOIFM5PI47YTEH7RJXXU63LYV55PETBK3MA4J6Q, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.337259024 (511h53m18.208342059s) [noise=true , address=45.91.116.4:4939 , piece_id=2H6SKUCMPPP2EJ5XJQWDM3JLXCXTVQD6Q3I3DJTGOZ2LMYIMP6LA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.337645880 (511h53m18.208728925s) [noise=true , address=45.91.116.6:4974 , piece_id=FJ3MQ7CY7AR3562LYRXLK7KD4LKZYQHNZTBU5BKMYUSDMUDOLKWA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.338028559 (511h53m18.209111604s) [noise=true , address=145.79.227.45:4231 , piece_id=AT2QYUKSLZKM7G6MAEI7KACJOYHKKIEPHYSLHPY54R3DAJ3QZBKA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.338555312 (511h53m18.209638347s) [noise=true , address=45.91.116.11:4119 , piece_id=GSGUPXCACFV3QJQCKCYYPE4AJAUVGXZPUTUIAOR3IBO5QXJBHGXQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.339045595 (511h53m18.21012863s) [noise=true , address=45.91.116.5:4297 , piece_id=O4O5Y5Q6DXH4EDMRQGROV6B6RJ4QDPSFMC5HZPA3KXJLHPBMX2NA, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.340153275 (511h53m18.21123632s) [noise=true , address=145.79.227.3:4149 , piece_id=IR36U33D2GHCKU4QE7ZNMSXQFDNGLFXRN25YVM7JJ67EMJ5U47OQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.340378865 (511h53m18.2114619s) [noise=true , address=45.91.116.10:4081 , piece_id=OCWPQZMLEQLOUXPTMKZWNAQXN5AYVZ4CKHFYCLPW7QSH7X7UZI4Q, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.341511361 (511h53m18.212594396s) [noise=true , address=145.79.227.48:4301 , piece_id=DGFM47GJEQI77SFLKOV7RR26LQI3OBS6YQAMDZ4QKJV645TQST7A, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.341545345 (511h53m18.21262838s) [noise=true , address=45.91.116.8:4047 , piece_id=C4EHKQYIRN7B4RA6RDOXXQXTV6HL43K6FDWTZGHNGEMLOIXA5RZQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.343464640 (511h53m18.214547685s) [noise=true , address=145.79.227.39:4014 , piece_id=MZFKWP3BSEXAHB5U47QSFETWNCPMKSHVVYVUVAHK773ZGNRZIFEQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:18.346131518 (511h53m18.217214554s) [noise=true , address=45.91.116.9:4263 , piece_id=UVWN6C3RWYWQJCDYIERJAUA4F76D7EQTXH5GHCJAUMNA4MVN5QMQ, seg_pos={0 0}, upload=449174]: Uploading piece... Dec 26 23:22:20.056143369 (511h53m19.927226404s) [upload=449174]: Next returned nil segment Dec 26 23:22:20.056161033 (511h53m19.927244078s) [upload=449174]: Waiting for error group managing segments... Dec 26 23:22:20.341483688 (511h53m20.212566723s) [noise=true , address=145.79.227.48:4301 , piece_id=DGFM47GJEQI77SFLKOV7RR26LQI3OBS6YQAMDZ4QKJV645TQST7A, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.346708866 (511h53m20.217791901s) [noise=true , address=145.79.227.45:4231 , piece_id=AT2QYUKSLZKM7G6MAEI7KACJOYHKKIEPHYSLHPY54R3DAJ3QZBKA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.351752971 (511h53m20.222836016s) [noise=true , address=145.79.227.47:4266 , piece_id=LFBF4PKKFEWHYQRGXUEHMWTVHS26OKSAZXWF5YAHJLIRFEUBRKOA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.352979848 (511h53m20.224062893s) [noise=true , address=145.79.227.3:4149 , piece_id=IR36U33D2GHCKU4QE7ZNMSXQFDNGLFXRN25YVM7JJ67EMJ5U47OQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.354083539 (511h53m20.225166594s) [noise=true , address=145.79.227.39:4014 , piece_id=MZFKWP3BSEXAHB5U47QSFETWNCPMKSHVVYVUVAHK773ZGNRZIFEQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.367548749 (511h53m20.238631784s) [noise=true , address=145.79.227.37:4566 , piece_id=CNSCGTMCFEHWELZONKUNC2TEJFNIAH6TC4XWUEZB7A5YW2TTVWOQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.369340511 (511h53m20.240423546s) [noise=true , address=145.79.227.29:4943 , piece_id=25IQXVLTMAOLS3K2VAZ4SL5ECLZ3BPNJTXOPDNM47QTXHX3M5HVA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.562842385 (511h53m20.43392542s) [noise=true , address=45.91.116.6:4974 , piece_id=FJ3MQ7CY7AR3562LYRXLK7KD4LKZYQHNZTBU5BKMYUSDMUDOLKWA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.587894568 (511h53m20.458977603s) [noise=true , address=45.91.116.9:4263 , piece_id=UVWN6C3RWYWQJCDYIERJAUA4F76D7EQTXH5GHCJAUMNA4MVN5QMQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.623504484 (511h53m20.494587529s) [noise=true , address=45.91.116.11:4119 , piece_id=GSGUPXCACFV3QJQCKCYYPE4AJAUVGXZPUTUIAOR3IBO5QXJBHGXQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.647454999 (511h53m20.518538034s) [noise=true , address=45.91.116.5:4297 , piece_id=O4O5Y5Q6DXH4EDMRQGROV6B6RJ4QDPSFMC5HZPA3KXJLHPBMX2NA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.650344832 (511h53m20.521427877s) [noise=true , address=45.91.116.4:4939 , piece_id=2H6SKUCMPPP2EJ5XJQWDM3JLXCXTVQD6Q3I3DJTGOZ2LMYIMP6LA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.694666056 (511h53m20.565749091s) [noise=true , address=45.91.116.8:4047 , piece_id=C4EHKQYIRN7B4RA6RDOXXQXTV6HL43K6FDWTZGHNGEMLOIXA5RZQ, seg_pos={0 0}, upload=449174]: Done uploading piece. err: <nil> Dec 26 23:22:20.694686314 (511h53m20.565769349s) [seg_pos={0 0}, upload=449174]: Segment reached optimal threshold of 13 pieces. Dec 26 23:22:20.694876687 (511h53m20.565959722s) [noise=true , address=145.79.227.33:4869 , piece_id=XMFA6KKJV35XYM2JTRBKQBQLHVTTHFCNRC2ZAESBT5FRFES64NRA, seg_pos={0 0}, upload=449174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 26 23:22:20.694882929 (511h53m20.565965964s) [noise=true , address=145.79.227.33:4869 , piece_id=XMFA6KKJV35XYM2JTRBKQBQLHVTTHFCNRC2ZAESBT5FRFES64NRA, seg_pos={0 0}, upload=449174]: Long tail cancellation due to optimal threshold Dec 26 23:22:20.695113168 (511h53m20.566196213s) [noise=true , address=45.91.116.7:4010 , piece_id=ILPUYVGX4T6UEGOIFM5PI47YTEH7RJXXU63LYV55PETBK3MA4J6Q, seg_pos={0 0}, upload=449174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 23:22:20.695141471 (511h53m20.566224506s) [noise=true , address=45.91.116.7:4010 , piece_id=ILPUYVGX4T6UEGOIFM5PI47YTEH7RJXXU63LYV55PETBK3MA4J6Q, seg_pos={0 0}, upload=449174]: Long tail cancellation due to optimal threshold Dec 26 23:22:20.695170616 (511h53m20.566253691s) [noise=true , address=45.91.116.10:4081 , piece_id=OCWPQZMLEQLOUXPTMKZWNAQXN5AYVZ4CKHFYCLPW7QSH7X7UZI4Q, seg_pos={0 0}, upload=449174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 23:22:20.695181157 (511h53m20.566264202s) [noise=true , address=45.91.116.10:4081 , piece_id=OCWPQZMLEQLOUXPTMKZWNAQXN5AYVZ4CKHFYCLPW7QSH7X7UZI4Q, seg_pos={0 0}, upload=449174]: Long tail cancellation due to optimal threshold Dec 26 23:22:20.695204391 (511h53m20.566287426s) [seg_pos={0 0}, upload=449174]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:22:20.695278321 (511h53m20.566361356s) [upload=449174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:22:20.695285386 (511h53m20.566368441s) [upload=449174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:22:20.980288329 (511h53m20.851371374s) [upload=449174]: Done uploading segments... Dec 26 23:22:20.980312145 (511h53m20.85139518s) [upload=449174]: Upload finished. err: <nil> Duration: 30.002193757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.031776486s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:22:44 UTC) orbiter warning: context deadline exceeded Duration: 6.031804759s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.925403724s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:21:37 UTC) orbiter warning: context deadline exceeded Duration: 5.925426617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X....X.......X.......X..............................X.X.X.X.] duration: 13.125659035s trace: https://jaeger.dev.storj.io/trace/337ba9c61edd965f FAIL (2025-12-26 23:21:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6D5I3KTXVIQ5LOV42ETTAHVPV4W4VE5TEGOHNGUAKP2CNDVVRRHQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4QSYAT26DQS2IQ6QCRRY4JEO7QUMKAHDMOVLLGR2KEXKPWEEXC5A) context deadline exceeded ======LOGS====== Dec 26 23:20:50.701949029 (511h51m50.573032064s) [upload=449151]: Starting upload Dec 26 23:20:50.702024434 (511h51m50.573107469s) [upload=449151]: Done starting upload Dec 26 23:20:50.702095920 (511h51m50.573178955s) [upload=449151]: Uploading segments... Dec 26 23:20:50.702267677 (511h51m50.573350712s) [upload=449151]: Got next segment. Inline: false Dec 26 23:20:50.702298374 (511h51m50.573381419s) [upload=449151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:20:50.702304737 (511h51m50.573387772s) [upload=449151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:20:50.979064225 (511h51m50.85014726s) [seg_pos={0 0}, upload=449151]: Begin upload segment... Dec 26 23:20:50.979118458 (511h51m50.850201493s) [seg_pos={0 0}, upload=449151]: Done begin upload segment. Dec 26 23:20:50.979157112 (511h51m50.850240147s) [noise=true , address=45.91.116.13:4154 , piece_id=F2PDP2HDLYB2XYFVYIYWOOTBGP5UXUUQ3E6KPYLC7FKV5KG4IZDA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.979182259 (511h51m50.850265294s) [noise=true , address=145.79.227.17:4971 , piece_id=VAPUPOD6YHXJETFS7M6YD4LUPHUTL54A4AYK6THIOJDCPFGHRXVQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.979606508 (511h51m50.850689543s) [noise=true , address=77.90.31.21:4880 , piece_id=6XMAERWAFT4O4GSIKPNG3OKDLCFEIXBY7YE6FQFMBO6T36BCHUQA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.980014624 (511h51m50.851097659s) [noise=true , address=77.90.31.13:4627 , piece_id=7CQJX2UQ7Q3OEFD7MEIP6ZOKA6A2LEFWFK2TD5F23FPJA2K4KTJA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.980407483 (511h51m50.851490538s) [noise=true , address=145.79.227.12:4767 , piece_id=TA3M775TF22HVQLVW2EWVTPPCJWQ3KMNNLHLNOCCPCGVPRFYXIHA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.980829526 (511h51m50.851912571s) [noise=true , address=45.91.116.8:4048 , piece_id=6D5I3KTXVIQ5LOV42ETTAHVPV4W4VE5TEGOHNGUAKP2CNDVVRRHQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.981277169 (511h51m50.852360214s) [noise=true , address=45.91.116.14:4189 , piece_id=4QSYAT26DQS2IQ6QCRRY4JEO7QUMKAHDMOVLLGR2KEXKPWEEXC5A, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.981754798 (511h51m50.852837843s) [noise=true , address=145.79.227.45:4230 , piece_id=VHE6FOSSXH5HYKNZ7DFH43253RZMDB477VBQTEOJ34V3TGA7CZRA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.981982502 (511h51m50.853065547s) [noise=true , address=145.79.227.47:4267 , piece_id=U3JFX4FFY5ECMVO62PTWYZBYASCBEG25RRNHRUI4HNUGCRG66DQA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.982409154 (511h51m50.853492199s) [noise=true , address=145.79.227.16:4912 , piece_id=JNFRHNSXPWKLNQCEEROBG4NCTXRMUHCFC4SUWUORESCXAMYF24HQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.982625405 (511h51m50.853708451s) [noise=true , address=145.79.227.48:4303 , piece_id=IXSVRT7AEIFPVDUVMOJX6B4HW3QS6MFEBW7D7JJ4CTN4CBNGXWYA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.982930176 (511h51m50.854013221s) [noise=true , address=145.79.227.33:4869 , piece_id=QPMQRDLPGBOT2XIX7UWKSE66VITA22TGUFOKVOZPUFXMIU62BSNA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983174591 (511h51m50.854257637s) [noise=true , address=145.79.227.50:4375 , piece_id=ULGGUTKH2ZA5KR53VYEW3YHU7RGVBZPLSNCJQIDPIEXUR42HRBZA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983413507 (511h51m50.854496552s) [noise=true , address=145.79.227.46:4603 , piece_id=ZTYPVN43BSA27EW3UPCZOTPBKXHCEQVHBKW3VVNQSLKXPXB2EJOA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983836121 (511h51m50.854919166s) [noise=true , address=45.91.116.4:4939 , piece_id=FZ65TMZSD6PHCFP4XQMCGQALTOFXHZXUZOASIEPAXTFNIH3NU3MQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983890024 (511h51m50.854973059s) [noise=true , address=145.79.227.37:4567 , piece_id=Z5OB6Y5IEUUX5LQC3VQPWFRYKCBX5ZBDKLE57CDTHJCAI5GV2VVA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:52.693433524 (511h51m52.564516569s) [upload=449151]: Next returned nil segment Dec 26 23:20:52.693452009 (511h51m52.564535044s) [upload=449151]: Waiting for error group managing segments... Dec 26 23:20:52.986966020 (511h51m52.858049055s) [noise=true , address=145.79.227.12:4767 , piece_id=TA3M775TF22HVQLVW2EWVTPPCJWQ3KMNNLHLNOCCPCGVPRFYXIHA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.987432499 (511h51m52.858515534s) [noise=true , address=145.79.227.37:4567 , piece_id=Z5OB6Y5IEUUX5LQC3VQPWFRYKCBX5ZBDKLE57CDTHJCAI5GV2VVA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.987482093 (511h51m52.858565128s) [noise=true , address=145.79.227.48:4303 , piece_id=IXSVRT7AEIFPVDUVMOJX6B4HW3QS6MFEBW7D7JJ4CTN4CBNGXWYA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.987900299 (511h51m52.858983344s) [noise=true , address=145.79.227.50:4375 , piece_id=ULGGUTKH2ZA5KR53VYEW3YHU7RGVBZPLSNCJQIDPIEXUR42HRBZA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.991341151 (511h51m52.862424196s) [noise=true , address=145.79.227.46:4603 , piece_id=ZTYPVN43BSA27EW3UPCZOTPBKXHCEQVHBKW3VVNQSLKXPXB2EJOA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.992258307 (511h51m52.863341352s) [noise=true , address=145.79.227.17:4971 , piece_id=VAPUPOD6YHXJETFS7M6YD4LUPHUTL54A4AYK6THIOJDCPFGHRXVQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.996910406 (511h51m52.867993441s) [noise=true , address=145.79.227.45:4230 , piece_id=VHE6FOSSXH5HYKNZ7DFH43253RZMDB477VBQTEOJ34V3TGA7CZRA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.006298041 (511h51m52.877381076s) [noise=true , address=145.79.227.47:4267 , piece_id=U3JFX4FFY5ECMVO62PTWYZBYASCBEG25RRNHRUI4HNUGCRG66DQA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.032333456 (511h51m52.903416491s) [noise=true , address=145.79.227.16:4912 , piece_id=JNFRHNSXPWKLNQCEEROBG4NCTXRMUHCFC4SUWUORESCXAMYF24HQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.215952892 (511h51m53.087035938s) [noise=true , address=45.91.116.8:4048 , piece_id=6D5I3KTXVIQ5LOV42ETTAHVPV4W4VE5TEGOHNGUAKP2CNDVVRRHQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.232239996 (511h51m53.103323031s) [noise=true , address=45.91.116.14:4189 , piece_id=4QSYAT26DQS2IQ6QCRRY4JEO7QUMKAHDMOVLLGR2KEXKPWEEXC5A, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.235603600 (511h51m53.106686635s) [noise=true , address=77.90.31.13:4627 , piece_id=7CQJX2UQ7Q3OEFD7MEIP6ZOKA6A2LEFWFK2TD5F23FPJA2K4KTJA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.242433636 (511h51m53.113516681s) [noise=true , address=77.90.31.21:4880 , piece_id=6XMAERWAFT4O4GSIKPNG3OKDLCFEIXBY7YE6FQFMBO6T36BCHUQA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.242455297 (511h51m53.113538332s) [seg_pos={0 0}, upload=449151]: Segment reached optimal threshold of 13 pieces. Dec 26 23:20:53.242667491 (511h51m53.113750536s) [noise=true , address=145.79.227.33:4869 , piece_id=QPMQRDLPGBOT2XIX7UWKSE66VITA22TGUFOKVOZPUFXMIU62BSNA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 26 23:20:53.242675717 (511h51m53.113758752s) [noise=true , address=145.79.227.33:4869 , piece_id=QPMQRDLPGBOT2XIX7UWKSE66VITA22TGUFOKVOZPUFXMIU62BSNA, seg_pos={0 0}, upload=449151]: Long tail cancellation due to optimal threshold Dec 26 23:20:53.242727174 (511h51m53.113810219s) [noise=true , address=45.91.116.13:4154 , piece_id=F2PDP2HDLYB2XYFVYIYWOOTBGP5UXUUQ3E6KPYLC7FKV5KG4IZDA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 23:20:53.242735130 (511h51m53.113818165s) [noise=true , address=45.91.116.13:4154 , piece_id=F2PDP2HDLYB2XYFVYIYWOOTBGP5UXUUQ3E6KPYLC7FKV5KG4IZDA, seg_pos={0 0}, upload=449151]: Long tail cancellation due to optimal threshold Dec 26 23:20:53.242879705 (511h51m53.11396274s) [noise=true , address=45.91.116.4:4939 , piece_id=FZ65TMZSD6PHCFP4XQMCGQALTOFXHZXUZOASIEPAXTFNIH3NU3MQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 23:20:53.242886788 (511h51m53.113969834s) [noise=true , address=45.91.116.4:4939 , piece_id=FZ65TMZSD6PHCFP4XQMCGQALTOFXHZXUZOASIEPAXTFNIH3NU3MQ, seg_pos={0 0}, upload=449151]: Long tail cancellation due to optimal threshold Dec 26 23:20:53.242902488 (511h51m53.113985523s) [seg_pos={0 0}, upload=449151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:20:53.242984946 (511h51m53.114067991s) [upload=449151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:20:53.242991197 (511h51m53.114074242s) [upload=449151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:20:53.549134987 (511h51m53.420218022s) [upload=449151]: Done uploading segments... Dec 26 23:20:53.549153793 (511h51m53.420236828s) [upload=449151]: Upload finished. err: <nil> Duration: 13.12567296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X....X.......X.......X..............................X.X.X.X] duration: 30.005383781s trace: https://jaeger.dev.storj.io/trace/337ba9c61edd965f FAIL (2025-12-26 23:21:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 6D5I3KTXVIQ5LOV42ETTAHVPV4W4VE5TEGOHNGUAKP2CNDVVRRHQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4QSYAT26DQS2IQ6QCRRY4JEO7QUMKAHDMOVLLGR2KEXKPWEEXC5A) context deadline exceeded ======LOGS====== Dec 26 23:20:50.701949029 (511h51m50.573032064s) [upload=449151]: Starting upload Dec 26 23:20:50.702024434 (511h51m50.573107469s) [upload=449151]: Done starting upload Dec 26 23:20:50.702095920 (511h51m50.573178955s) [upload=449151]: Uploading segments... Dec 26 23:20:50.702267677 (511h51m50.573350712s) [upload=449151]: Got next segment. Inline: false Dec 26 23:20:50.702298374 (511h51m50.573381419s) [upload=449151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:20:50.702304737 (511h51m50.573387772s) [upload=449151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:20:50.979064225 (511h51m50.85014726s) [seg_pos={0 0}, upload=449151]: Begin upload segment... Dec 26 23:20:50.979118458 (511h51m50.850201493s) [seg_pos={0 0}, upload=449151]: Done begin upload segment. Dec 26 23:20:50.979157112 (511h51m50.850240147s) [noise=true , address=45.91.116.13:4154 , piece_id=F2PDP2HDLYB2XYFVYIYWOOTBGP5UXUUQ3E6KPYLC7FKV5KG4IZDA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.979182259 (511h51m50.850265294s) [noise=true , address=145.79.227.17:4971 , piece_id=VAPUPOD6YHXJETFS7M6YD4LUPHUTL54A4AYK6THIOJDCPFGHRXVQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.979606508 (511h51m50.850689543s) [noise=true , address=77.90.31.21:4880 , piece_id=6XMAERWAFT4O4GSIKPNG3OKDLCFEIXBY7YE6FQFMBO6T36BCHUQA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.980014624 (511h51m50.851097659s) [noise=true , address=77.90.31.13:4627 , piece_id=7CQJX2UQ7Q3OEFD7MEIP6ZOKA6A2LEFWFK2TD5F23FPJA2K4KTJA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.980407483 (511h51m50.851490538s) [noise=true , address=145.79.227.12:4767 , piece_id=TA3M775TF22HVQLVW2EWVTPPCJWQ3KMNNLHLNOCCPCGVPRFYXIHA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.980829526 (511h51m50.851912571s) [noise=true , address=45.91.116.8:4048 , piece_id=6D5I3KTXVIQ5LOV42ETTAHVPV4W4VE5TEGOHNGUAKP2CNDVVRRHQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.981277169 (511h51m50.852360214s) [noise=true , address=45.91.116.14:4189 , piece_id=4QSYAT26DQS2IQ6QCRRY4JEO7QUMKAHDMOVLLGR2KEXKPWEEXC5A, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.981754798 (511h51m50.852837843s) [noise=true , address=145.79.227.45:4230 , piece_id=VHE6FOSSXH5HYKNZ7DFH43253RZMDB477VBQTEOJ34V3TGA7CZRA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.981982502 (511h51m50.853065547s) [noise=true , address=145.79.227.47:4267 , piece_id=U3JFX4FFY5ECMVO62PTWYZBYASCBEG25RRNHRUI4HNUGCRG66DQA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.982409154 (511h51m50.853492199s) [noise=true , address=145.79.227.16:4912 , piece_id=JNFRHNSXPWKLNQCEEROBG4NCTXRMUHCFC4SUWUORESCXAMYF24HQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.982625405 (511h51m50.853708451s) [noise=true , address=145.79.227.48:4303 , piece_id=IXSVRT7AEIFPVDUVMOJX6B4HW3QS6MFEBW7D7JJ4CTN4CBNGXWYA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.982930176 (511h51m50.854013221s) [noise=true , address=145.79.227.33:4869 , piece_id=QPMQRDLPGBOT2XIX7UWKSE66VITA22TGUFOKVOZPUFXMIU62BSNA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983174591 (511h51m50.854257637s) [noise=true , address=145.79.227.50:4375 , piece_id=ULGGUTKH2ZA5KR53VYEW3YHU7RGVBZPLSNCJQIDPIEXUR42HRBZA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983413507 (511h51m50.854496552s) [noise=true , address=145.79.227.46:4603 , piece_id=ZTYPVN43BSA27EW3UPCZOTPBKXHCEQVHBKW3VVNQSLKXPXB2EJOA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983836121 (511h51m50.854919166s) [noise=true , address=45.91.116.4:4939 , piece_id=FZ65TMZSD6PHCFP4XQMCGQALTOFXHZXUZOASIEPAXTFNIH3NU3MQ, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:50.983890024 (511h51m50.854973059s) [noise=true , address=145.79.227.37:4567 , piece_id=Z5OB6Y5IEUUX5LQC3VQPWFRYKCBX5ZBDKLE57CDTHJCAI5GV2VVA, seg_pos={0 0}, upload=449151]: Uploading piece... Dec 26 23:20:52.693433524 (511h51m52.564516569s) [upload=449151]: Next returned nil segment Dec 26 23:20:52.693452009 (511h51m52.564535044s) [upload=449151]: Waiting for error group managing segments... Dec 26 23:20:52.986966020 (511h51m52.858049055s) [noise=true , address=145.79.227.12:4767 , piece_id=TA3M775TF22HVQLVW2EWVTPPCJWQ3KMNNLHLNOCCPCGVPRFYXIHA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.987432499 (511h51m52.858515534s) [noise=true , address=145.79.227.37:4567 , piece_id=Z5OB6Y5IEUUX5LQC3VQPWFRYKCBX5ZBDKLE57CDTHJCAI5GV2VVA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.987482093 (511h51m52.858565128s) [noise=true , address=145.79.227.48:4303 , piece_id=IXSVRT7AEIFPVDUVMOJX6B4HW3QS6MFEBW7D7JJ4CTN4CBNGXWYA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.987900299 (511h51m52.858983344s) [noise=true , address=145.79.227.50:4375 , piece_id=ULGGUTKH2ZA5KR53VYEW3YHU7RGVBZPLSNCJQIDPIEXUR42HRBZA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.991341151 (511h51m52.862424196s) [noise=true , address=145.79.227.46:4603 , piece_id=ZTYPVN43BSA27EW3UPCZOTPBKXHCEQVHBKW3VVNQSLKXPXB2EJOA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.992258307 (511h51m52.863341352s) [noise=true , address=145.79.227.17:4971 , piece_id=VAPUPOD6YHXJETFS7M6YD4LUPHUTL54A4AYK6THIOJDCPFGHRXVQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:52.996910406 (511h51m52.867993441s) [noise=true , address=145.79.227.45:4230 , piece_id=VHE6FOSSXH5HYKNZ7DFH43253RZMDB477VBQTEOJ34V3TGA7CZRA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.006298041 (511h51m52.877381076s) [noise=true , address=145.79.227.47:4267 , piece_id=U3JFX4FFY5ECMVO62PTWYZBYASCBEG25RRNHRUI4HNUGCRG66DQA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.032333456 (511h51m52.903416491s) [noise=true , address=145.79.227.16:4912 , piece_id=JNFRHNSXPWKLNQCEEROBG4NCTXRMUHCFC4SUWUORESCXAMYF24HQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.215952892 (511h51m53.087035938s) [noise=true , address=45.91.116.8:4048 , piece_id=6D5I3KTXVIQ5LOV42ETTAHVPV4W4VE5TEGOHNGUAKP2CNDVVRRHQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.232239996 (511h51m53.103323031s) [noise=true , address=45.91.116.14:4189 , piece_id=4QSYAT26DQS2IQ6QCRRY4JEO7QUMKAHDMOVLLGR2KEXKPWEEXC5A, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.235603600 (511h51m53.106686635s) [noise=true , address=77.90.31.13:4627 , piece_id=7CQJX2UQ7Q3OEFD7MEIP6ZOKA6A2LEFWFK2TD5F23FPJA2K4KTJA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.242433636 (511h51m53.113516681s) [noise=true , address=77.90.31.21:4880 , piece_id=6XMAERWAFT4O4GSIKPNG3OKDLCFEIXBY7YE6FQFMBO6T36BCHUQA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: <nil> Dec 26 23:20:53.242455297 (511h51m53.113538332s) [seg_pos={0 0}, upload=449151]: Segment reached optimal threshold of 13 pieces. Dec 26 23:20:53.242667491 (511h51m53.113750536s) [noise=true , address=145.79.227.33:4869 , piece_id=QPMQRDLPGBOT2XIX7UWKSE66VITA22TGUFOKVOZPUFXMIU62BSNA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 26 23:20:53.242675717 (511h51m53.113758752s) [noise=true , address=145.79.227.33:4869 , piece_id=QPMQRDLPGBOT2XIX7UWKSE66VITA22TGUFOKVOZPUFXMIU62BSNA, seg_pos={0 0}, upload=449151]: Long tail cancellation due to optimal threshold Dec 26 23:20:53.242727174 (511h51m53.113810219s) [noise=true , address=45.91.116.13:4154 , piece_id=F2PDP2HDLYB2XYFVYIYWOOTBGP5UXUUQ3E6KPYLC7FKV5KG4IZDA, seg_pos={0 0}, upload=449151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 23:20:53.242735130 (511h51m53.113818165s) [noise=true , address=45.91.116.13:4154 , piece_id=F2PDP2HDLYB2XYFVYIYWOOTBGP5UXUUQ3E6KPYLC7FKV5KG4IZDA, seg_pos={0 0}, upload=449151]: Long tail cancellation due to optimal threshold Dec 26 23:20:53.242879705 (511h51m53.11396274s) [noise=true , address=45.91.116.4:4939 , piece_id=FZ65TMZSD6PHCFP4XQMCGQALTOFXHZXUZOASIEPAXTFNIH3NU3MQ, seg_pos={0 0}, upload=449151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 23:20:53.242886788 (511h51m53.113969834s) [noise=true , address=45.91.116.4:4939 , piece_id=FZ65TMZSD6PHCFP4XQMCGQALTOFXHZXUZOASIEPAXTFNIH3NU3MQ, seg_pos={0 0}, upload=449151]: Long tail cancellation due to optimal threshold Dec 26 23:20:53.242902488 (511h51m53.113985523s) [seg_pos={0 0}, upload=449151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:20:53.242984946 (511h51m53.114067991s) [upload=449151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:20:53.242991197 (511h51m53.114074242s) [upload=449151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:20:53.549134987 (511h51m53.420218022s) [upload=449151]: Done uploading segments... Dec 26 23:20:53.549153793 (511h51m53.420236828s) [upload=449151]: Upload finished. err: <nil> Duration: 30.005395113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....X.......X.......X..............................X.X.X.] duration: 24.489033418s trace: https://jaeger.dev.storj.io/trace/136f097503400aab FAIL (2025-12-26 23:20:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA) context deadline exceeded ======LOGS====== 23:19:12.466635016 (511h50m12.337718061s) [upload=449126]: Starting upload Dec 26 23:19:12.466708205 (511h50m12.337791251s) [upload=449126]: Done starting upload Dec 26 23:19:12.466748342 (511h50m12.337831377s) [upload=449126]: Uploading segments... Dec 26 23:19:12.466977268 (511h50m12.338060313s) [upload=449126]: Got next segment. Inline: false Dec 26 23:19:12.467022674 (511h50m12.338105709s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:19:12.467028906 (511h50m12.338111951s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:19:12.718488249 (511h50m12.589571284s) [seg_pos={0 0}, upload=449126]: Begin upload segment... Dec 26 23:19:12.718549737 (511h50m12.589632772s) [seg_pos={0 0}, upload=449126]: Done begin upload segment. Dec 26 23:19:12.718612126 (511h50m12.589695161s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.718695524 (511h50m12.589778569s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719430825 (511h50m12.59051386s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719925256 (511h50m12.591008291s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720102744 (511h50m12.591185779s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720347370 (511h50m12.591430405s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720454173 (511h50m12.591537208s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720783159 (511h50m12.591866194s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720877379 (511h50m12.591960414s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721148295 (511h50m12.59223133s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721424371 (511h50m12.592507406s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721612629 (511h50m12.592695664s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722022259 (511h50m12.593105304s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722125967 (511h50m12.593209002s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722612243 (511h50m12.593695278s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722780423 (511h50m12.593863458s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:14.424582260 (511h50m14.295665305s) [upload=449126]: Next returned nil segment Dec 26 23:19:14.424595757 (511h50m14.295678802s) [upload=449126]: Waiting for error group managing segments... Dec 26 23:19:14.706470205 (511h50m14.57755324s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712004413 (511h50m14.583087448s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712235883 (511h50m14.583318918s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.717901832 (511h50m14.588984867s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.726128848 (511h50m14.597211893s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.744914238 (511h50m14.615997273s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.746678788 (511h50m14.617761823s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.762938268 (511h50m14.634021323s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.912235681 (511h50m14.783318726s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.121372597 (511h50m14.992455642s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.179091482 (511h50m15.050174527s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.643867068 (511h50m15.514950113s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762665928 (511h50m15.633748963s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762688852 (511h50m15.633771897s) [seg_pos={0 0}, upload=449126]: Segment reached optimal threshold of 13 pieces. Dec 26 23:19:15.762917777 (511h50m15.634000822s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.762933358 (511h50m15.634016393s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.762959066 (511h50m15.634042111s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection Dec 26 23:19:15.762968254 (511h50m15.634051289s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763020493 (511h50m15.634103528s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.763026755 (511h50m15.63410979s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763044008 (511h50m15.634127043s) [seg_pos={0 0}, upload=449126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:19:15.763113721 (511h50m15.634196756s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:19:15.763118520 (511h50m15.634201565s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:19:16.065024811 (511h50m15.936107866s) [upload=449126]: Done uploading segments... Dec 26 23:19:16.065046683 (511h50m15.936129718s) [upload=449126]: Upload finished. err: <nil> Duration: 24.489058425s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.932181859s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:20:38 UTC) orbiter warning: context deadline exceeded Duration: 5.932207828s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.8089082s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:19:51 UTC) orbiter warning: context deadline exceeded Duration: 5.808940372s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....X.......X.......X..............................X.X.X] duration: 30.002142281s trace: https://jaeger.dev.storj.io/trace/136f097503400aab FAIL (2025-12-26 23:19:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA) context deadline exceeded ======LOGS====== 23:19:12.466635016 (511h50m12.337718061s) [upload=449126]: Starting upload Dec 26 23:19:12.466708205 (511h50m12.337791251s) [upload=449126]: Done starting upload Dec 26 23:19:12.466748342 (511h50m12.337831377s) [upload=449126]: Uploading segments... Dec 26 23:19:12.466977268 (511h50m12.338060313s) [upload=449126]: Got next segment. Inline: false Dec 26 23:19:12.467022674 (511h50m12.338105709s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:19:12.467028906 (511h50m12.338111951s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:19:12.718488249 (511h50m12.589571284s) [seg_pos={0 0}, upload=449126]: Begin upload segment... Dec 26 23:19:12.718549737 (511h50m12.589632772s) [seg_pos={0 0}, upload=449126]: Done begin upload segment. Dec 26 23:19:12.718612126 (511h50m12.589695161s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.718695524 (511h50m12.589778569s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719430825 (511h50m12.59051386s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.719925256 (511h50m12.591008291s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720102744 (511h50m12.591185779s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720347370 (511h50m12.591430405s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720454173 (511h50m12.591537208s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720783159 (511h50m12.591866194s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.720877379 (511h50m12.591960414s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721148295 (511h50m12.59223133s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721424371 (511h50m12.592507406s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.721612629 (511h50m12.592695664s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722022259 (511h50m12.593105304s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722125967 (511h50m12.593209002s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722612243 (511h50m12.593695278s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:12.722780423 (511h50m12.593863458s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Uploading piece... Dec 26 23:19:14.424582260 (511h50m14.295665305s) [upload=449126]: Next returned nil segment Dec 26 23:19:14.424595757 (511h50m14.295678802s) [upload=449126]: Waiting for error group managing segments... Dec 26 23:19:14.706470205 (511h50m14.57755324s) [noise=true , address=145.79.227.50:4373 , piece_id=ZU2D5OTWQDJICRQHBQBGN25E2PSCGJ7UI255IJEYEE365O6FRQLA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712004413 (511h50m14.583087448s) [noise=true , address=145.79.227.51:4410 , piece_id=TBQOX3KZDEYCN4OJCIKDTRTYZG3U63GRNOR2VASS5YS6IS2Z3SMQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.712235883 (511h50m14.583318918s) [noise=true , address=145.79.227.38:4977 , piece_id=P2R7BOAXHPQA2KV4F3VFXCVI42IQPF6PI4NP436SPTEN3OXMRAEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.717901832 (511h50m14.588984867s) [noise=true , address=145.79.227.11:4732 , piece_id=C2ARAE6EVIJN3BKIO73XSWS4ODN74GNNYNMIHFRW2JMGB2NAWQFA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.726128848 (511h50m14.597211893s) [noise=true , address=145.79.227.37:4565 , piece_id=OLCTEG2CU2X5IZVAWNKGRTO7G5EDN6UCC3UYIV3HP62OELSFX7GA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.744914238 (511h50m14.615997273s) [noise=true , address=145.79.227.45:4229 , piece_id=HZISC42PXKHME4QD4KWEXUZJNFDGJ2T4TXZ3TY6AGAOK6VC5KVWA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.746678788 (511h50m14.617761823s) [noise=true , address=145.79.227.5:4223 , piece_id=S3U2GOSLFCVL6NXVGUVSEUBKR7N746LNMJUAL2H7SAMLEISR2DEA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.762938268 (511h50m14.634021323s) [noise=true , address=145.79.227.46:4601 , piece_id=HYME7ERIU5Z62M3K4JVYXHS3L7LPTCOPYSKHYQDZEOZ4CPJXHXEQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:14.912235681 (511h50m14.783318726s) [noise=true , address=77.90.31.16:4736 , piece_id=BIDIVS73X6IL4GAVPMG4IDGFBH5MICS2GXVBR422V6NLHGT7QCNA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.121372597 (511h50m14.992455642s) [noise=true , address=45.91.116.6:4974 , piece_id=HBNDBNN5ESS3RPZBTZIWWPM4WSQC6HDAZLURIBRJTNCXBDZ5DYVQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.179091482 (511h50m15.050174527s) [noise=true , address=45.91.116.13:4156 , piece_id=GGMBC45WPA67JJIMQYCOLZGMNKDDEEOWHDQWWJHZIVU33MVIPD2A, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.643867068 (511h50m15.514950113s) [noise=true , address=77.90.31.17:4773 , piece_id=3PEZ5BAJWFVCB327WFLTZ63FBKCUSDSHVLJBPAYPWRXHOVPZS2PA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762665928 (511h50m15.633748963s) [noise=true , address=77.90.31.13:4629 , piece_id=NUQHUVXGSUMMZT576QYY5YWYQ7RNGHOZC2MOAAZYY7RERNMS2TJA, seg_pos={0 0}, upload=449126]: Done uploading piece. err: <nil> Dec 26 23:19:15.762688852 (511h50m15.633771897s) [seg_pos={0 0}, upload=449126]: Segment reached optimal threshold of 13 pieces. Dec 26 23:19:15.762917777 (511h50m15.634000822s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.762933358 (511h50m15.634016393s) [noise=true , address=45.91.116.7:4011 , piece_id=T7GCLPLI74HDUC74Y26Q6ANIUVWYAQBZIMYPV44LLP5GQ7YIZH5Q, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.762959066 (511h50m15.634042111s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41168->145.79.227.7:4295: use of closed network connection Dec 26 23:19:15.762968254 (511h50m15.634051289s) [noise=true , address=145.79.227.7:4295 , piece_id=3WJKNLCOXUSMWKV3ASL3VLZHD4MPCNZPTYEEFZNYO7SRGKKLNGRQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763020493 (511h50m15.634103528s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 23:19:15.763026755 (511h50m15.63410979s) [noise=true , address=145.79.227.43:4158 , piece_id=MZ4ZH44R7PPUPXGNQPAGVNPA2ARZPCO3Z6VALITWHIRLRBNAXXIQ, seg_pos={0 0}, upload=449126]: Long tail cancellation due to optimal threshold Dec 26 23:19:15.763044008 (511h50m15.634127043s) [seg_pos={0 0}, upload=449126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:19:15.763113721 (511h50m15.634196756s) [upload=449126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:19:15.763118520 (511h50m15.634201565s) [upload=449126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:19:16.065024811 (511h50m15.936107866s) [upload=449126]: Done uploading segments... Dec 26 23:19:16.065046683 (511h50m15.936129718s) [upload=449126]: Upload finished. err: <nil> Duration: 30.002165044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X....X.......X.......X..............................X.X.] duration: 6.062299474s trace: https://jaeger.dev.storj.io/trace/671df6d0e83b79ba FAIL (2025-12-26 23:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA) context deadline exceeded ======LOGS====== Dec 26 23:17:44.491436204 (511h48m44.362519239s) [upload=449107]: Starting upload Dec 26 23:17:44.491497711 (511h48m44.362580746s) [upload=449107]: Done starting upload Dec 26 23:17:44.491591670 (511h48m44.362674705s) [upload=449107]: Uploading segments... Dec 26 23:17:44.491687372 (511h48m44.362770407s) [upload=449107]: Got next segment. Inline: false Dec 26 23:17:44.491738199 (511h48m44.362821234s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:17:44.491745623 (511h48m44.362828658s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:17:44.742342985 (511h48m44.61342603s) [seg_pos={0 0}, upload=449107]: Begin upload segment... Dec 26 23:17:44.742384525 (511h48m44.61346757s) [seg_pos={0 0}, upload=449107]: Done begin upload segment. Dec 26 23:17:44.742411195 (511h48m44.61349423s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742415975 (511h48m44.61349901s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742821496 (511h48m44.613904541s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.743653751 (511h48m44.614736786s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744041980 (511h48m44.615125015s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744649828 (511h48m44.615732863s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744813028 (511h48m44.615896073s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744985236 (511h48m44.616068271s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745843140 (511h48m44.616926175s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745958499 (511h48m44.617041545s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.747158485 (511h48m44.61824154s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.748134744 (511h48m44.619217789s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.749173332 (511h48m44.620256377s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.751251518 (511h48m44.622334553s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.752908283 (511h48m44.623991318s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.757383205 (511h48m44.62846626s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:46.459493209 (511h48m46.330576244s) [upload=449107]: Next returned nil segment Dec 26 23:17:46.459507356 (511h48m46.330590391s) [upload=449107]: Waiting for error group managing segments... Dec 26 23:17:46.751409548 (511h48m46.622492594s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761303899 (511h48m46.632386954s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761498961 (511h48m46.632582006s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765415678 (511h48m46.636498713s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765491764 (511h48m46.636574799s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.782774852 (511h48m46.653857887s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783704623 (511h48m46.654787658s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783994234 (511h48m46.655077269s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.787077045 (511h48m46.65816008s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.800716205 (511h48m46.67179924s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.009597595 (511h48m46.88068063s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.040843763 (511h48m46.911926808s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046454385 (511h48m46.91753743s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046471699 (511h48m46.917554734s) [seg_pos={0 0}, upload=449107]: Segment reached optimal threshold of 13 pieces. Dec 26 23:17:47.046727105 (511h48m46.91781015s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046736051 (511h48m46.917819086s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.046969005 (511h48m46.91805204s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046975277 (511h48m46.918058312s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047225994 (511h48m46.918309029s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.047236405 (511h48m46.91831945s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047259197 (511h48m46.918342233s) [seg_pos={0 0}, upload=449107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:17:47.047368827 (511h48m46.918451862s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:17:47.047374959 (511h48m46.918458004s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:17:47.350840189 (511h48m47.221923234s) [upload=449107]: Done uploading segments... Dec 26 23:17:47.350858774 (511h48m47.221941809s) [upload=449107]: Upload finished. err: <nil> Duration: 6.062330021s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.836973718s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:18:41 UTC) orbiter warning: context deadline exceeded Duration: 5.836997984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X....X.......X.......X..............................X.X] duration: 30.001772348s trace: https://jaeger.dev.storj.io/trace/671df6d0e83b79ba FAIL (2025-12-26 23:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA) context deadline exceeded ======LOGS====== Dec 26 23:17:44.491436204 (511h48m44.362519239s) [upload=449107]: Starting upload Dec 26 23:17:44.491497711 (511h48m44.362580746s) [upload=449107]: Done starting upload Dec 26 23:17:44.491591670 (511h48m44.362674705s) [upload=449107]: Uploading segments... Dec 26 23:17:44.491687372 (511h48m44.362770407s) [upload=449107]: Got next segment. Inline: false Dec 26 23:17:44.491738199 (511h48m44.362821234s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:17:44.491745623 (511h48m44.362828658s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:17:44.742342985 (511h48m44.61342603s) [seg_pos={0 0}, upload=449107]: Begin upload segment... Dec 26 23:17:44.742384525 (511h48m44.61346757s) [seg_pos={0 0}, upload=449107]: Done begin upload segment. Dec 26 23:17:44.742411195 (511h48m44.61349423s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742415975 (511h48m44.61349901s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.742821496 (511h48m44.613904541s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.743653751 (511h48m44.614736786s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744041980 (511h48m44.615125015s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744649828 (511h48m44.615732863s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744813028 (511h48m44.615896073s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.744985236 (511h48m44.616068271s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745843140 (511h48m44.616926175s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.745958499 (511h48m44.617041545s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.747158485 (511h48m44.61824154s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.748134744 (511h48m44.619217789s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.749173332 (511h48m44.620256377s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.751251518 (511h48m44.622334553s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.752908283 (511h48m44.623991318s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:44.757383205 (511h48m44.62846626s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Uploading piece... Dec 26 23:17:46.459493209 (511h48m46.330576244s) [upload=449107]: Next returned nil segment Dec 26 23:17:46.459507356 (511h48m46.330590391s) [upload=449107]: Waiting for error group managing segments... Dec 26 23:17:46.751409548 (511h48m46.622492594s) [noise=true , address=145.79.227.48:4301 , piece_id=CW4SM5SZPCLFJ3AENH6QNSZEKAD7ZJNO7WG4JSV6IHNBKGX7A2PA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761303899 (511h48m46.632386954s) [noise=true , address=145.79.227.45:4229 , piece_id=ZGGG32VDXT76S7P2CVMNIR5LU4IQ6XPKC3K3CRCAJRSO6HTUME6Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.761498961 (511h48m46.632582006s) [noise=true , address=145.79.227.39:4014 , piece_id=6CBYURI5XNZ7Z4XOM47LEUOONRRKEOIAZUG2H4JY4KDRYG4ANFTQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765415678 (511h48m46.636498713s) [noise=true , address=145.79.227.5:4221 , piece_id=MALZPJVFO6ICWSHH3AZLJZY5H67VKIYYLPYN7G74ZVAQR62KTBSA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.765491764 (511h48m46.636574799s) [noise=true , address=145.79.227.37:4565 , piece_id=OQGE7BPBUTZ7WFMAHB57FLANU4OFNGAMUV6V2CKNYVZPHPCYTMMA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.782774852 (511h48m46.653857887s) [noise=true , address=145.79.227.42:4122 , piece_id=FLJF66ZEKF7GYHHQWLUMMHQHWXNIU6BWSBWP2G4J6PODGGHCO26Q, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783704623 (511h48m46.654787658s) [noise=true , address=145.79.227.13:4804 , piece_id=PPQJUBGJPENZOSVAYCV2VK26NK3BZOWOGB47DCBFQDFMFDWQ4JOQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.783994234 (511h48m46.655077269s) [noise=true , address=145.79.227.11:4732 , piece_id=CSCOBQMIUA5VTK7FVHXBFTLMHIV4BFQ5ZT4WEFT6Y57XOIF7F6HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.787077045 (511h48m46.65816008s) [noise=true , address=145.79.227.46:4602 , piece_id=S4SNS3KE2SAIW54QMEH6UUMXRKQET3TUWXCU5R77ZRNCIIYG4JUA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:46.800716205 (511h48m46.67179924s) [noise=true , address=145.79.227.12:4765 , piece_id=QCDW2SVVZWPIUHUPKFP7HYPVC2IMWTIRBI5FR2INT6QIPAT235HQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.009597595 (511h48m46.88068063s) [noise=true , address=45.91.116.6:4976 , piece_id=LOCUEZQOQ6OH7KCRXL4OJVEYIYAEAEJLY6XJP3DGBNOZBIRVVPYA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.040843763 (511h48m46.911926808s) [noise=true , address=45.91.116.5:4299 , piece_id=AJ3422X3RNUAPJ6VVNY75LQMV6CHFLI45P6TCW36M6IGMBONFWKQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046454385 (511h48m46.91753743s) [noise=true , address=45.91.116.14:4190 , piece_id=QF6ZAHTPCPNBDETIZ23XLHVXNCPCUHYLKYQPB3E6NTY47G5JCZCA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: <nil> Dec 26 23:17:47.046471699 (511h48m46.917554734s) [seg_pos={0 0}, upload=449107]: Segment reached optimal threshold of 13 pieces. Dec 26 23:17:47.046727105 (511h48m46.91781015s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046736051 (511h48m46.917819086s) [noise=true , address=45.91.116.4:4937 , piece_id=ZZSI3MJWIOG6FEHB3A6TJ34RK2LOLXEG7UAAQLYMFB7LDZWUJDIQ, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.046969005 (511h48m46.91805204s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.046975277 (511h48m46.918058312s) [noise=true , address=45.91.116.9:4261 , piece_id=JVDKOZXH6HVPD7BSGKARQTEMQADKD2J7AS2JJAVGJ45IKKPGBBLA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047225994 (511h48m46.918309029s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 23:17:47.047236405 (511h48m46.91831945s) [noise=true , address=45.91.116.7:4011 , piece_id=KCIYMORIJFMSSXTI4DOEJI7GXTCOR5TFILRMYFIAIZVV7PJVBVIA, seg_pos={0 0}, upload=449107]: Long tail cancellation due to optimal threshold Dec 26 23:17:47.047259197 (511h48m46.918342233s) [seg_pos={0 0}, upload=449107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:17:47.047368827 (511h48m46.918451862s) [upload=449107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:17:47.047374959 (511h48m46.918458004s) [upload=449107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:17:47.350840189 (511h48m47.221923234s) [upload=449107]: Done uploading segments... Dec 26 23:17:47.350858774 (511h48m47.221941809s) [upload=449107]: Upload finished. err: <nil> Duration: 30.00179973s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.856806568s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:17:27 UTC) orbiter warning: context deadline exceeded Duration: 5.85683412s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.235429477s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:17:02 UTC) orbiter warning: context deadline exceeded Duration: 6.235454556s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....X....X.......X.......X..............................X.] duration: 6.172218754s trace: https://jaeger.dev.storj.io/trace/130eb89f4e53f1e0 FAIL (2025-12-26 23:16:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA) context deadline exceeded ======LOGS====== 449086]: Starting upload Dec 26 23:16:03.827516883 (511h47m3.698599928s) [upload=449086]: Done starting upload Dec 26 23:16:03.827664724 (511h47m3.698747769s) [upload=449086]: Uploading segments... Dec 26 23:16:03.827810051 (511h47m3.698893086s) [upload=449086]: Got next segment. Inline: false Dec 26 23:16:03.827849877 (511h47m3.698932922s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:16:03.827854716 (511h47m3.698937751s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:16:04.076311273 (511h47m3.947394328s) [seg_pos={0 0}, upload=449086]: Begin upload segment... Dec 26 23:16:04.076387258 (511h47m3.947470303s) [seg_pos={0 0}, upload=449086]: Done begin upload segment. Dec 26 23:16:04.076424929 (511h47m3.947507974s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077198041 (511h47m3.948281086s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077396269 (511h47m3.948479304s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077886262 (511h47m3.948969297s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078070983 (511h47m3.949154019s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078379481 (511h47m3.949462526s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078766578 (511h47m3.949849623s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079195546 (511h47m3.950278581s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079657595 (511h47m3.95074064s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080066523 (511h47m3.951149558s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080149692 (511h47m3.951232727s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080585060 (511h47m3.951668106s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080969733 (511h47m3.952052778s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081353323 (511h47m3.952436368s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081787120 (511h47m3.952870165s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081961282 (511h47m3.953044327s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:05.800268217 (511h47m5.671351252s) [upload=449086]: Next returned nil segment Dec 26 23:16:05.800289818 (511h47m5.671372863s) [upload=449086]: Waiting for error group managing segments... Dec 26 23:16:06.086124055 (511h47m5.95720709s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.095285099 (511h47m5.966368135s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.096266187 (511h47m5.967349233s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099479436 (511h47m5.970562471s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099654379 (511h47m5.970737444s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.100010197 (511h47m5.971093242s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.102804658 (511h47m5.973887693s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.114909597 (511h47m5.985992642s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.327932642 (511h47m6.199015677s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.356727242 (511h47m6.227810287s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.361363350 (511h47m6.232446395s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.388538935 (511h47m6.25962198s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439308448 (511h47m6.310391483s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439329067 (511h47m6.310412102s) [seg_pos={0 0}, upload=449086]: Segment reached optimal threshold of 13 pieces. Dec 26 23:16:06.439539608 (511h47m6.310622643s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:16:06.439545950 (511h47m6.310628985s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439706997 (511h47m6.310790032s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection Dec 26 23:16:06.439722807 (511h47m6.310805842s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439903611 (511h47m6.310986646s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:56926->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 23:16:06.439911226 (511h47m6.310994261s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439927066 (511h47m6.311010111s) [seg_pos={0 0}, upload=449086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:16:06.439996107 (511h47m6.311079152s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:16:06.440001427 (511h47m6.311084472s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:16:06.737326335 (511h47m6.60840938s) [upload=449086]: Done uploading segments... Dec 26 23:16:06.737353497 (511h47m6.608436542s) [upload=449086]: Upload finished. err: <nil> Duration: 6.172247148s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.280162894s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:16:39 UTC) orbiter warning: context deadline exceeded Duration: 6.280187411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.....X....X.......X.......X..............................X] duration: 30.00172731s trace: https://jaeger.dev.storj.io/trace/130eb89f4e53f1e0 FAIL (2025-12-26 23:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA) context deadline exceeded ======LOGS====== 449086]: Starting upload Dec 26 23:16:03.827516883 (511h47m3.698599928s) [upload=449086]: Done starting upload Dec 26 23:16:03.827664724 (511h47m3.698747769s) [upload=449086]: Uploading segments... Dec 26 23:16:03.827810051 (511h47m3.698893086s) [upload=449086]: Got next segment. Inline: false Dec 26 23:16:03.827849877 (511h47m3.698932922s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 23:16:03.827854716 (511h47m3.698937751s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 23:16:04.076311273 (511h47m3.947394328s) [seg_pos={0 0}, upload=449086]: Begin upload segment... Dec 26 23:16:04.076387258 (511h47m3.947470303s) [seg_pos={0 0}, upload=449086]: Done begin upload segment. Dec 26 23:16:04.076424929 (511h47m3.947507974s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077198041 (511h47m3.948281086s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077396269 (511h47m3.948479304s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.077886262 (511h47m3.948969297s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078070983 (511h47m3.949154019s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078379481 (511h47m3.949462526s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.078766578 (511h47m3.949849623s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079195546 (511h47m3.950278581s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.079657595 (511h47m3.95074064s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080066523 (511h47m3.951149558s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080149692 (511h47m3.951232727s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080585060 (511h47m3.951668106s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.080969733 (511h47m3.952052778s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081353323 (511h47m3.952436368s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081787120 (511h47m3.952870165s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:04.081961282 (511h47m3.953044327s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Uploading piece... Dec 26 23:16:05.800268217 (511h47m5.671351252s) [upload=449086]: Next returned nil segment Dec 26 23:16:05.800289818 (511h47m5.671372863s) [upload=449086]: Waiting for error group managing segments... Dec 26 23:16:06.086124055 (511h47m5.95720709s) [noise=true , address=145.79.227.50:4373 , piece_id=PYWDDOIZMFWSODZHE2INI3VGFSUOCGSIK4MCTVET2VC563PJVLWA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.095285099 (511h47m5.966368135s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VRI5ZGK2XLDLT6MBPJUXXTF4JGLVBQFAALJUDSRL4YXAJKQ5ZA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.096266187 (511h47m5.967349233s) [noise=true , address=145.79.227.51:4409 , piece_id=NW3YMT4WJOTESBG4VLZMA5QLC6QBSMFQ6RUE7QJHRRQSDHUCTPIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099479436 (511h47m5.970562471s) [noise=true , address=145.79.227.52:4446 , piece_id=HIWZ5KJ5NS6E4OEU5THWMYXRSFGBLR5XRQTJQQKRUR6URZKXDVGQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.099654379 (511h47m5.970737444s) [noise=true , address=145.79.227.37:4567 , piece_id=PSOXFGFGPNSFWQRP2C3WOBI6ERTLA4K4AZNKS2WB2DKPAK6DWOPQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.100010197 (511h47m5.971093242s) [noise=true , address=145.79.227.49:4338 , piece_id=XUC5VF2KVI2M3WSXME6TTPJDTZD5WPWOLOJEQDD67W2QKDDW345A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.102804658 (511h47m5.973887693s) [noise=true , address=145.79.227.38:4978 , piece_id=3L5EOFRYYHYUGUAXH6LFCB7JLDNF3ASWMVFBCDF7Z7NMD3INHU5A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.114909597 (511h47m5.985992642s) [noise=true , address=145.79.227.47:4267 , piece_id=ANZOJFHTGGWU3HE24OHZ7YWF5M6ONZBVQVYFTWRDN6PWMUS2IQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.327932642 (511h47m6.199015677s) [noise=true , address=45.91.116.12:4228 , piece_id=BJHWKNTZFP7FJCGSDW5PK6QIQIFAWPM2YUHDIK4ULBKOCCRVPFIQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.356727242 (511h47m6.227810287s) [noise=true , address=45.91.116.4:4939 , piece_id=2LDMFLULUFKDEM7FRFMJP5STS6KDD6XAFCQJE7BFLCOT26ELAFAA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.361363350 (511h47m6.232446395s) [noise=true , address=45.91.116.8:4046 , piece_id=36QPCOF5BYCWH3FWKFYGTQYNYPITIHJYHEYHNUN74EV3CMUGHR7A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.388538935 (511h47m6.25962198s) [noise=true , address=45.91.116.5:4298 , piece_id=J4VPBFSGIYAFU3QJRNVK26XOPBVXCZZUHOTPRXITLNDXEFRVLH4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439308448 (511h47m6.310391483s) [noise=true , address=45.91.116.11:4119 , piece_id=SDSXPMCEU34QOVN35VFPVAAHUVCMWDS2SWUMAMCBX3ZZGCFLXEXA, seg_pos={0 0}, upload=449086]: Done uploading piece. err: <nil> Dec 26 23:16:06.439329067 (511h47m6.310412102s) [seg_pos={0 0}, upload=449086]: Segment reached optimal threshold of 13 pieces. Dec 26 23:16:06.439539608 (511h47m6.310622643s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 23:16:06.439545950 (511h47m6.310628985s) [noise=true , address=45.91.116.9:4263 , piece_id=GCFI63PMDS2MTX6WHFW3MKFPOTXAWVI6HQVWNVRUBEXZ6UVFUQNQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439706997 (511h47m6.310790032s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52480->45.91.116.7:4011: use of closed network connection Dec 26 23:16:06.439722807 (511h47m6.310805842s) [noise=true , address=45.91.116.7:4011 , piece_id=7W5ARDKVKREZSW7BDLI2NUQAMP2MTVKBVTNORKXRV44N5R3SCS4A, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439903611 (511h47m6.310986646s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:56926->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 23:16:06.439911226 (511h47m6.310994261s) [noise=true , address=77.90.31.16:4735 , piece_id=EDEVQWV4OXGD2I4I4OQ3OTU43UNPPNZGXC3KHONSLXTRGGGIXFMQ, seg_pos={0 0}, upload=449086]: Long tail cancellation due to optimal threshold Dec 26 23:16:06.439927066 (511h47m6.311010111s) [seg_pos={0 0}, upload=449086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 23:16:06.439996107 (511h47m6.311079152s) [upload=449086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 23:16:06.440001427 (511h47m6.311084472s) [upload=449086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 23:16:06.737326335 (511h47m6.60840938s) [upload=449086]: Done uploading segments... Dec 26 23:16:06.737353497 (511h47m6.608436542s) [upload=449086]: Upload finished. err: <nil> Duration: 30.001753541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.....X....X.......X.......X..............................] duration: 20.759702242s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 20.759734734s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.93010071s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:15:29 UTC) orbiter warning: context deadline exceeded Duration: 5.930129767s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.....X....X.......X.......X.............................] duration: 13.089817573s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 13.08984741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX.....X....X.......X.......X............................] duration: 26.792985765s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:14:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 26.793014128s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.151395349s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:14:17 UTC) orbiter warning: context deadline exceeded Duration: 6.151420387s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX.....X....X.......X.......X...........................] duration: 6.078206459s trace: https://jaeger.dev.storj.io/trace/7d87e7639a2182f9 FAIL (2025-12-26 23:13:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ) context deadline exceeded ======LOGS====== Dec 26 22:51:03.902506696 (511h22m3.773589731s) [upload=448722]: Starting upload Dec 26 22:51:03.902590806 (511h22m3.773673861s) [upload=448722]: Done starting upload Dec 26 22:51:03.902795997 (511h22m3.773879042s) [upload=448722]: Uploading segments... Dec 26 22:51:03.902953397 (511h22m3.774036432s) [upload=448722]: Got next segment. Inline: false Dec 26 22:51:03.902992612 (511h22m3.774075657s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:51:03.902999304 (511h22m3.774082349s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:51:04.149612448 (511h22m4.020695483s) [seg_pos={0 0}, upload=448722]: Begin upload segment... Dec 26 22:51:04.149669537 (511h22m4.020752582s) [seg_pos={0 0}, upload=448722]: Done begin upload segment. Dec 26 22:51:04.149690045 (511h22m4.02077309s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.149751863 (511h22m4.020834908s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151004599 (511h22m4.022087644s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151434438 (511h22m4.022517483s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.151883001 (511h22m4.022966046s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152321577 (511h22m4.023404622s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.152744883 (511h22m4.023827928s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.153175503 (511h22m4.024258538s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.154385396 (511h22m4.025468441s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155308645 (511h22m4.02639168s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.155697074 (511h22m4.026780119s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.156932998 (511h22m4.028016043s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.157251304 (511h22m4.028334349s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.158334917 (511h22m4.029417962s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.159371461 (511h22m4.030454496s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:04.160954185 (511h22m4.03203724s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Uploading piece... Dec 26 22:51:05.867335403 (511h22m5.738418448s) [upload=448722]: Next returned nil segment Dec 26 22:51:05.867349159 (511h22m5.738432194s) [upload=448722]: Waiting for error group managing segments... Dec 26 22:51:06.147005023 (511h22m6.018088058s) [noise=true , address=145.79.227.42:4121 , piece_id=SIODENBTBRVOYBFEFSMJLEVSGHRT6QMY2V3QTIKONZBPN7QJ73IA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.149166958 (511h22m6.020250003s) [noise=true , address=145.79.227.50:4373 , piece_id=XWDTBMAAX7TCFDXS6SNWROIMUNYUAQS5ASJEF2IVYQBREDZBGBHA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.161657432 (511h22m6.032740478s) [noise=true , address=145.79.227.49:4339 , piece_id=UQ4ENQOOHR73RARJVB4VORSZAJEUUDFNXD3ZXHTRGTSTOLCOXZJQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.163909320 (511h22m6.034992365s) [noise=true , address=145.79.227.52:4445 , piece_id=CKNIPP4D3KJ6O6RPQEOLEBK5YJZU5V6BO73CDTOOURI3GLNGOVEA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.166597870 (511h22m6.037680915s) [noise=true , address=145.79.227.36:4518 , piece_id=IUVFOR4RLYZFDGKIETKP6QVF5QRBVZH4RMZKHFXZ4Y2RHLSHOMKA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.173108867 (511h22m6.044191902s) [noise=true , address=145.79.227.48:4301 , piece_id=DIW25GUHLUQZLU4UPQJLYLIPRPFM3QJFUYH2V6RZAINIMLQNIA2Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196171231 (511h22m6.067254276s) [noise=true , address=145.79.227.47:4265 , piece_id=VDPUO2KBWRGDIXJFT6MGWSF3WVQZHFN4RFUYY5FNAHVS5K3GRJZQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.196396480 (511h22m6.067479525s) [noise=true , address=145.79.227.11:4730 , piece_id=DV3JV3Y24ABF5U4GFBCDEFW2ZI7B6OEG5OP6G2CADNPLGQKRMSGA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.319299624 (511h22m6.190382659s) [noise=true , address=145.79.227.51:4410 , piece_id=5NIVGFII4OSN5NM6BVBBR6N34NU5IMM7LIS2MK76HGTPVT2CFR2A, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.338654599 (511h22m6.209737634s) [noise=true , address=77.90.31.12:4590 , piece_id=JTWH53BASDSOYXZHB5IEPZUQBOGVFTQ4O6LTPOUJ3EO7B3NMXDKQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.387827761 (511h22m6.258910836s) [noise=true , address=45.91.116.6:4975 , piece_id=TOAWCGUQHGFIX7HEIE4MENMYKPTK26QLL3UWEJWGNTCFFZIXJ4MQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.410414329 (511h22m6.281497374s) [noise=true , address=45.91.116.5:4300 , piece_id=3HD6JJKNXS6IGWMS3T5OO5SYWX6P6PRHNAW44NIUDMAPWXXB2AYA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415351640 (511h22m6.286434675s) [noise=true , address=45.91.116.14:4191 , piece_id=FWDZ2RBLZGQPGIBSKOXPNUPKHLZMKCUKYXTN3KJM5UGYX4E2MAAA, seg_pos={0 0}, upload=448722]: Done uploading piece. err: <nil> Dec 26 22:51:06.415372670 (511h22m6.286455715s) [seg_pos={0 0}, upload=448722]: Segment reached optimal threshold of 13 pieces. Dec 26 22:51:06.415618488 (511h22m6.286701523s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50684->145.79.227.6:4259: use of closed network connection Dec 26 22:51:06.415625151 (511h22m6.286708186s) [noise=true , address=145.79.227.6:4259 , piece_id=4Z6SQMIZQPXB2TSKAOZJERJ5X4GUKTSHY6M6XY5AFX624NNQUDBQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415713409 (511h22m6.286796444s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415746502 (511h22m6.286829547s) [noise=true , address=45.91.116.9:4262 , piece_id=SOXF2MLYPLEHCN7JDKJP5DBCN77Z6UMHEAVE5I3MCITDOVS4TXVQ, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415907699 (511h22m6.286990734s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 22:51:06.415913901 (511h22m6.286996936s) [noise=true , address=45.91.116.4:4939 , piece_id=GQ3WKFEKVTJ2U2PXJAKILAFCK4WVHJBIGAGV3XNBT7WUSGTERY7Q, seg_pos={0 0}, upload=448722]: Long tail cancellation due to optimal threshold Dec 26 22:51:06.415933708 (511h22m6.287016743s) [seg_pos={0 0}, upload=448722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:51:06.416009042 (511h22m6.287092077s) [upload=448722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:51:06.416014843 (511h22m6.287097888s) [upload=448722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:51:06.702772418 (511h22m6.573855453s) [upload=448722]: Done uploading segments... Dec 26 22:51:06.702796564 (511h22m6.573879609s) [upload=448722]: Upload finished. err: <nil> Duration: 6.078239443s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.32825283s trace: https://jaeger.dev.storj.io/trace/f21975653b9b600 FAIL (2025-12-26 23:13:04 UTC) orbiter warning: context deadline exceeded Duration: 6.328429065s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)