Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X] duration: 5.001482428s trace: https://jaeger.dev.storj.io/trace/55bd066ce8df500d FAIL (2025-12-28 11:33:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001506543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X...X...XXX...X.XX.X..XX..X.......X.......................] duration: 6.018600767s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:32:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.01862864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X...X...XXX...X.XX.X..XX..X.......X......................] duration: 6.455861173s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:32:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.455891941s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....] duration: 3.674895566s trace: https://jaeger.dev.storj.io/trace/12fce2113c0ed882 FAIL (2025-12-28 11:32:16 UTC) orbiter warning: context deadline exceeded Duration: 3.674916096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X........X...........X..X...] duration: 35.511736402s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:32:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 35.51176792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X.X...X...XXX...X.XX.X..XX..X.......X.....................] duration: 6.224582035s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:31:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.224627402s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX....] duration: 3.404507511s trace: https://jaeger.dev.storj.io/trace/12fce2113c0ed882 FAIL (2025-12-28 11:31:24 UTC) orbiter warning: context deadline exceeded Duration: 3.404534483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X.X...X...XXX...X.XX.X..XX..X.......X....................] duration: 5.87715974s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:31:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.877228611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X........X...........X..X..] duration: 42.39466523s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:31:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 42.394690829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.X.X...X...XXX...X.XX.X..XX..X.......X...................] duration: 6.221009498s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:30:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.221070194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX...] duration: 4.259173549s trace: https://jaeger.dev.storj.io/trace/12fce2113c0ed882 FAIL (2025-12-28 11:30:42 UTC) orbiter warning: context deadline exceeded Duration: 4.259194598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X.X.X...X...XXX...X.XX.X..XX..X.......X..................] duration: 5.93074188s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.930761727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X........X...........X..X.] duration: 53.242354046s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:30:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 53.242367392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X.................] duration: 5.650191384s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.650219086s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX..] duration: 5.001469391s trace: https://jaeger.dev.storj.io/trace/12fce2113c0ed882 FAIL (2025-12-28 11:29:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001498648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X........X...........X..X] duration: 1m0.004922106s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:29:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 1m0.00493972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X................] duration: 5.998274239s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:29:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.998305999s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.] duration: 3.55322253s trace: https://jaeger.dev.storj.io/trace/12fce2113c0ed882 FAIL (2025-12-28 11:28:53 UTC) orbiter warning: context deadline exceeded Duration: 3.553248799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX] duration: 5.000959912s trace: https://jaeger.dev.storj.io/trace/12fce2113c0ed882 FAIL (2025-12-28 11:28:26 UTC) orbiter warning: context deadline exceeded Duration: 5.000983467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X...............] duration: 6.065992679s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.066019271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X........X...........X..] duration: 39.313376286s trace: https://jaeger.dev.storj.io/trace/2ce1f4cfb1cfb7ef FAIL (2025-12-28 11:28:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.11:4745 , piece_id=VDOGZCCCSGHNURRMJ74WGGFBNJC6PQCFZQFQ57DW7TL7U3NI5GNQ, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.620815664 (547h56m34.491898699s) [noise=true , address=145.79.227.43:4174 , piece_id=AKMC6XVLDVSFXSMX6EBKM2BULGQYS4PUYR6MNUQXPGFV43I73YFA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.621212800 (547h56m34.492295846s) [noise=true , address=77.90.31.15:4708 , piece_id=WI5BBWEG53LKOJIV6PVHKWIRTKURJPVZXNCSGQYEA3FERSIEZ7UA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.621629364 (547h56m34.492712409s) [noise=true , address=145.79.227.44:4204 , piece_id=RZXLRWR7H2YOYPO2HF476QQNIDGLKCWESRNLOJGUCMWTM3UDIQLA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.622116411 (547h56m34.493199446s) [noise=true , address=77.90.31.14:4674 , piece_id=BI5O2ZQMR4SU4DMWS5MAITBOG32KF7QVAZS5WYN45M6LGRXQTDRA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.622726412 (547h56m34.493809447s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.623166521 (547h56m34.494249566s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.624338111 (547h56m34.495421156s) [noise=true , address=145.79.227.40:4077 , piece_id=XZADB7VPNQBZE6P6XZDPGCRGTLEDJP7ZQHSTXEE7DCMOUGZ4QO5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.624668931 (547h56m34.495751976s) [noise=true , address=77.90.31.10:4533 , piece_id=EJ4BXYL3VHGJDQSWTRBK2MDIMBYZWVGPNGVBAW5CXATDNIG3SO3A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.625409591 (547h56m34.496492626s) [noise=true , address=145.79.227.30:4790 , piece_id=AJJSRQJJQQBQZQYKKGJXB3AZMBGNXM5CSVOPUGTX2QMPDRTDQG4A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.626726179 (547h56m34.497809214s) [noise=true , address=193.149.164.143:4695 , piece_id=XQVRYNRGL6TK6QJ6UUH6DYCLUZ5JDXRC4TTBWZ3I3BMRSXBMFA5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.627082006 (547h56m34.498165041s) [noise=true , address=45.91.116.10:4096 , piece_id=ITZILOHFHA4COSK5RQZPDVSUK3AHSVOKEJPTRUV53KYLLQHMLICA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.629519978 (547h56m34.500603013s) [noise=true , address=145.79.227.46:4613 , piece_id=PKGBMUVJ7EDLOEFECMFKWR45HGBD6H5COQLBZT2GUZBZG4OOXVUA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.630836425 (547h56m34.50191946s) [noise=true , address=77.90.31.19:4869 , piece_id=YGORBKU5FW7EQW2IT2754N52B5LZCSBINVZN6BH65TM7UUBOEI5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.632054755 (547h56m34.50313779s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.633338991 (547h56m34.504422026s) [noise=true , address=77.90.31.13:4660 , piece_id=4JYN22KDY7PEM6UT3RGTMRGHADRRMF4ICKWCI66ZSJIUSGJB73DA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:35.932729840 (547h56m35.803812875s) [upload=480458]: Next returned nil segment Dec 28 11:25:35.932749927 (547h56m35.803832962s) [upload=480458]: Waiting for error group managing segments... Dec 28 11:25:36.263262379 (547h56m36.134345424s) [noise=true , address=193.149.164.143:4695 , piece_id=XQVRYNRGL6TK6QJ6UUH6DYCLUZ5JDXRC4TTBWZ3I3BMRSXBMFA5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.490162414 (547h56m36.361245449s) [noise=true , address=145.79.227.33:4878 , piece_id=YB7TH7MUPPMTXSD3TXH55Y2BS3JLZ5YT35SKLKK72BGRZOO6QMUQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.491849427 (547h56m36.362932462s) [noise=true , address=145.79.227.43:4174 , piece_id=AKMC6XVLDVSFXSMX6EBKM2BULGQYS4PUYR6MNUQXPGFV43I73YFA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.523702000 (547h56m36.394785045s) [noise=true , address=145.79.227.11:4745 , piece_id=VDOGZCCCSGHNURRMJ74WGGFBNJC6PQCFZQFQ57DW7TL7U3NI5GNQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.539014337 (547h56m36.410097373s) [noise=true , address=145.79.227.46:4613 , piece_id=PKGBMUVJ7EDLOEFECMFKWR45HGBD6H5COQLBZT2GUZBZG4OOXVUA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.703251933 (547h56m36.574334978s) [noise=true , address=77.90.31.10:4533 , piece_id=EJ4BXYL3VHGJDQSWTRBK2MDIMBYZWVGPNGVBAW5CXATDNIG3SO3A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.774268671 (547h56m36.645351706s) [noise=true , address=77.90.31.14:4674 , piece_id=BI5O2ZQMR4SU4DMWS5MAITBOG32KF7QVAZS5WYN45M6LGRXQTDRA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.792078694 (547h56m36.663161729s) [noise=true , address=77.90.31.19:4869 , piece_id=YGORBKU5FW7EQW2IT2754N52B5LZCSBINVZN6BH65TM7UUBOEI5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.821706180 (547h56m36.692789215s) [noise=true , address=45.91.116.10:4096 , piece_id=ITZILOHFHA4COSK5RQZPDVSUK3AHSVOKEJPTRUV53KYLLQHMLICA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.880056288 (547h56m36.751139323s) [noise=true , address=77.90.31.15:4708 , piece_id=WI5BBWEG53LKOJIV6PVHKWIRTKURJPVZXNCSGQYEA3FERSIEZ7UA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.006605499 (547h56m36.877688534s) [noise=true , address=77.90.31.13:4660 , piece_id=4JYN22KDY7PEM6UT3RGTMRGHADRRMF4ICKWCI66ZSJIUSGJB73DA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.373015596 (547h56m37.244098631s) [noise=true , address=145.79.227.40:4077 , piece_id=XZADB7VPNQBZE6P6XZDPGCRGTLEDJP7ZQHSTXEE7DCMOUGZ4QO5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.449754611 (547h56m37.320837646s) [noise=true , address=145.79.227.44:4204 , piece_id=RZXLRWR7H2YOYPO2HF476QQNIDGLKCWESRNLOJGUCMWTM3UDIQLA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.484055044 (547h56m37.355138079s) [noise=true , address=145.79.227.30:4790 , piece_id=AJJSRQJJQQBQZQYKKGJXB3AZMBGNXM5CSVOPUGTX2QMPDRTDQG4A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.484076976 (547h56m37.355160021s) [seg_pos={0 0}, upload=480458]: Segment reached optimal threshold of 14 pieces. Dec 28 11:25:37.484306723 (547h56m37.355389788s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484316702 (547h56m37.355399737s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484408347 (547h56m37.355491382s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484421281 (547h56m37.355504346s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484463993 (547h56m37.355547028s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection Dec 28 11:25:37.484472289 (547h56m37.355555334s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484625149 (547h56m37.355708194s) [noise=true , address=145.79.227.49:4364 , piece_id=BU3SILKDTQ5GDMER3UFVQXELR4TTO7NTOFF3AHEBOVNYWFSLWINQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484632975 (547h56m37.35571601s) [noise=true , address=145.79.227.49:4364 , piece_id=BU3SILKDTQ5GDMER3UFVQXELR4TTO7NTOFF3AHEBOVNYWFSLWINQ, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484725450 (547h56m37.355808496s) [seg_pos={0 0}, upload=480458]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:25:37.484809191 (547h56m37.355892226s) [upload=480458]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:25:37.484815603 (547h56m37.355898638s) [upload=480458]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:25:37.774194368 (547h56m37.645277413s) [upload=480458]: Done uploading segments... Dec 28 11:25:37.774217130 (547h56m37.645300176s) [upload=480458]: Upload finished. err: <nil> Duration: 39.313400253s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.....XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.X] duration: 5.00151984s trace: https://jaeger.dev.storj.io/trace/5ea19f57e162343a FAIL (2025-12-28 11:28:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001544186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X..............] duration: 5.611592356s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:27:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.611620379s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.] duration: 4.041637342s trace: https://jaeger.dev.storj.io/trace/673982ee27c668a7 FAIL (2025-12-28 11:27:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.041664766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X........X...........X.] duration: 47.747080076s trace: https://jaeger.dev.storj.io/trace/2ce1f4cfb1cfb7ef FAIL (2025-12-28 11:27:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.11:4745 , piece_id=VDOGZCCCSGHNURRMJ74WGGFBNJC6PQCFZQFQ57DW7TL7U3NI5GNQ, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.620815664 (547h56m34.491898699s) [noise=true , address=145.79.227.43:4174 , piece_id=AKMC6XVLDVSFXSMX6EBKM2BULGQYS4PUYR6MNUQXPGFV43I73YFA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.621212800 (547h56m34.492295846s) [noise=true , address=77.90.31.15:4708 , piece_id=WI5BBWEG53LKOJIV6PVHKWIRTKURJPVZXNCSGQYEA3FERSIEZ7UA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.621629364 (547h56m34.492712409s) [noise=true , address=145.79.227.44:4204 , piece_id=RZXLRWR7H2YOYPO2HF476QQNIDGLKCWESRNLOJGUCMWTM3UDIQLA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.622116411 (547h56m34.493199446s) [noise=true , address=77.90.31.14:4674 , piece_id=BI5O2ZQMR4SU4DMWS5MAITBOG32KF7QVAZS5WYN45M6LGRXQTDRA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.622726412 (547h56m34.493809447s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.623166521 (547h56m34.494249566s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.624338111 (547h56m34.495421156s) [noise=true , address=145.79.227.40:4077 , piece_id=XZADB7VPNQBZE6P6XZDPGCRGTLEDJP7ZQHSTXEE7DCMOUGZ4QO5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.624668931 (547h56m34.495751976s) [noise=true , address=77.90.31.10:4533 , piece_id=EJ4BXYL3VHGJDQSWTRBK2MDIMBYZWVGPNGVBAW5CXATDNIG3SO3A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.625409591 (547h56m34.496492626s) [noise=true , address=145.79.227.30:4790 , piece_id=AJJSRQJJQQBQZQYKKGJXB3AZMBGNXM5CSVOPUGTX2QMPDRTDQG4A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.626726179 (547h56m34.497809214s) [noise=true , address=193.149.164.143:4695 , piece_id=XQVRYNRGL6TK6QJ6UUH6DYCLUZ5JDXRC4TTBWZ3I3BMRSXBMFA5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.627082006 (547h56m34.498165041s) [noise=true , address=45.91.116.10:4096 , piece_id=ITZILOHFHA4COSK5RQZPDVSUK3AHSVOKEJPTRUV53KYLLQHMLICA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.629519978 (547h56m34.500603013s) [noise=true , address=145.79.227.46:4613 , piece_id=PKGBMUVJ7EDLOEFECMFKWR45HGBD6H5COQLBZT2GUZBZG4OOXVUA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.630836425 (547h56m34.50191946s) [noise=true , address=77.90.31.19:4869 , piece_id=YGORBKU5FW7EQW2IT2754N52B5LZCSBINVZN6BH65TM7UUBOEI5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.632054755 (547h56m34.50313779s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.633338991 (547h56m34.504422026s) [noise=true , address=77.90.31.13:4660 , piece_id=4JYN22KDY7PEM6UT3RGTMRGHADRRMF4ICKWCI66ZSJIUSGJB73DA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:35.932729840 (547h56m35.803812875s) [upload=480458]: Next returned nil segment Dec 28 11:25:35.932749927 (547h56m35.803832962s) [upload=480458]: Waiting for error group managing segments... Dec 28 11:25:36.263262379 (547h56m36.134345424s) [noise=true , address=193.149.164.143:4695 , piece_id=XQVRYNRGL6TK6QJ6UUH6DYCLUZ5JDXRC4TTBWZ3I3BMRSXBMFA5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.490162414 (547h56m36.361245449s) [noise=true , address=145.79.227.33:4878 , piece_id=YB7TH7MUPPMTXSD3TXH55Y2BS3JLZ5YT35SKLKK72BGRZOO6QMUQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.491849427 (547h56m36.362932462s) [noise=true , address=145.79.227.43:4174 , piece_id=AKMC6XVLDVSFXSMX6EBKM2BULGQYS4PUYR6MNUQXPGFV43I73YFA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.523702000 (547h56m36.394785045s) [noise=true , address=145.79.227.11:4745 , piece_id=VDOGZCCCSGHNURRMJ74WGGFBNJC6PQCFZQFQ57DW7TL7U3NI5GNQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.539014337 (547h56m36.410097373s) [noise=true , address=145.79.227.46:4613 , piece_id=PKGBMUVJ7EDLOEFECMFKWR45HGBD6H5COQLBZT2GUZBZG4OOXVUA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.703251933 (547h56m36.574334978s) [noise=true , address=77.90.31.10:4533 , piece_id=EJ4BXYL3VHGJDQSWTRBK2MDIMBYZWVGPNGVBAW5CXATDNIG3SO3A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.774268671 (547h56m36.645351706s) [noise=true , address=77.90.31.14:4674 , piece_id=BI5O2ZQMR4SU4DMWS5MAITBOG32KF7QVAZS5WYN45M6LGRXQTDRA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.792078694 (547h56m36.663161729s) [noise=true , address=77.90.31.19:4869 , piece_id=YGORBKU5FW7EQW2IT2754N52B5LZCSBINVZN6BH65TM7UUBOEI5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.821706180 (547h56m36.692789215s) [noise=true , address=45.91.116.10:4096 , piece_id=ITZILOHFHA4COSK5RQZPDVSUK3AHSVOKEJPTRUV53KYLLQHMLICA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.880056288 (547h56m36.751139323s) [noise=true , address=77.90.31.15:4708 , piece_id=WI5BBWEG53LKOJIV6PVHKWIRTKURJPVZXNCSGQYEA3FERSIEZ7UA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.006605499 (547h56m36.877688534s) [noise=true , address=77.90.31.13:4660 , piece_id=4JYN22KDY7PEM6UT3RGTMRGHADRRMF4ICKWCI66ZSJIUSGJB73DA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.373015596 (547h56m37.244098631s) [noise=true , address=145.79.227.40:4077 , piece_id=XZADB7VPNQBZE6P6XZDPGCRGTLEDJP7ZQHSTXEE7DCMOUGZ4QO5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.449754611 (547h56m37.320837646s) [noise=true , address=145.79.227.44:4204 , piece_id=RZXLRWR7H2YOYPO2HF476QQNIDGLKCWESRNLOJGUCMWTM3UDIQLA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.484055044 (547h56m37.355138079s) [noise=true , address=145.79.227.30:4790 , piece_id=AJJSRQJJQQBQZQYKKGJXB3AZMBGNXM5CSVOPUGTX2QMPDRTDQG4A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.484076976 (547h56m37.355160021s) [seg_pos={0 0}, upload=480458]: Segment reached optimal threshold of 14 pieces. Dec 28 11:25:37.484306723 (547h56m37.355389788s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484316702 (547h56m37.355399737s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484408347 (547h56m37.355491382s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484421281 (547h56m37.355504346s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484463993 (547h56m37.355547028s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection Dec 28 11:25:37.484472289 (547h56m37.355555334s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484625149 (547h56m37.355708194s) [noise=true , address=145.79.227.49:4364 , piece_id=BU3SILKDTQ5GDMER3UFVQXELR4TTO7NTOFF3AHEBOVNYWFSLWINQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484632975 (547h56m37.35571601s) [noise=true , address=145.79.227.49:4364 , piece_id=BU3SILKDTQ5GDMER3UFVQXELR4TTO7NTOFF3AHEBOVNYWFSLWINQ, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484725450 (547h56m37.355808496s) [seg_pos={0 0}, upload=480458]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:25:37.484809191 (547h56m37.355892226s) [upload=480458]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:25:37.484815603 (547h56m37.355898638s) [upload=480458]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:25:37.774194368 (547h56m37.645277413s) [upload=480458]: Done uploading segments... Dec 28 11:25:37.774217130 (547h56m37.645300176s) [upload=480458]: Upload finished. err: <nil> Duration: 47.747095735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X.............] duration: 5.747018169s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:27:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.747054026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X............] duration: 6.01657873s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:26:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.016595473s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X] duration: 5.001581022s trace: https://jaeger.dev.storj.io/trace/673982ee27c668a7 FAIL (2025-12-28 11:26:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001605858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X........X...........X] duration: 1m0.003291255s trace: https://jaeger.dev.storj.io/trace/2ce1f4cfb1cfb7ef FAIL (2025-12-28 11:26:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.11:4745 , piece_id=VDOGZCCCSGHNURRMJ74WGGFBNJC6PQCFZQFQ57DW7TL7U3NI5GNQ, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.620815664 (547h56m34.491898699s) [noise=true , address=145.79.227.43:4174 , piece_id=AKMC6XVLDVSFXSMX6EBKM2BULGQYS4PUYR6MNUQXPGFV43I73YFA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.621212800 (547h56m34.492295846s) [noise=true , address=77.90.31.15:4708 , piece_id=WI5BBWEG53LKOJIV6PVHKWIRTKURJPVZXNCSGQYEA3FERSIEZ7UA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.621629364 (547h56m34.492712409s) [noise=true , address=145.79.227.44:4204 , piece_id=RZXLRWR7H2YOYPO2HF476QQNIDGLKCWESRNLOJGUCMWTM3UDIQLA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.622116411 (547h56m34.493199446s) [noise=true , address=77.90.31.14:4674 , piece_id=BI5O2ZQMR4SU4DMWS5MAITBOG32KF7QVAZS5WYN45M6LGRXQTDRA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.622726412 (547h56m34.493809447s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.623166521 (547h56m34.494249566s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.624338111 (547h56m34.495421156s) [noise=true , address=145.79.227.40:4077 , piece_id=XZADB7VPNQBZE6P6XZDPGCRGTLEDJP7ZQHSTXEE7DCMOUGZ4QO5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.624668931 (547h56m34.495751976s) [noise=true , address=77.90.31.10:4533 , piece_id=EJ4BXYL3VHGJDQSWTRBK2MDIMBYZWVGPNGVBAW5CXATDNIG3SO3A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.625409591 (547h56m34.496492626s) [noise=true , address=145.79.227.30:4790 , piece_id=AJJSRQJJQQBQZQYKKGJXB3AZMBGNXM5CSVOPUGTX2QMPDRTDQG4A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.626726179 (547h56m34.497809214s) [noise=true , address=193.149.164.143:4695 , piece_id=XQVRYNRGL6TK6QJ6UUH6DYCLUZ5JDXRC4TTBWZ3I3BMRSXBMFA5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.627082006 (547h56m34.498165041s) [noise=true , address=45.91.116.10:4096 , piece_id=ITZILOHFHA4COSK5RQZPDVSUK3AHSVOKEJPTRUV53KYLLQHMLICA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.629519978 (547h56m34.500603013s) [noise=true , address=145.79.227.46:4613 , piece_id=PKGBMUVJ7EDLOEFECMFKWR45HGBD6H5COQLBZT2GUZBZG4OOXVUA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.630836425 (547h56m34.50191946s) [noise=true , address=77.90.31.19:4869 , piece_id=YGORBKU5FW7EQW2IT2754N52B5LZCSBINVZN6BH65TM7UUBOEI5A, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.632054755 (547h56m34.50313779s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:34.633338991 (547h56m34.504422026s) [noise=true , address=77.90.31.13:4660 , piece_id=4JYN22KDY7PEM6UT3RGTMRGHADRRMF4ICKWCI66ZSJIUSGJB73DA, seg_pos={0 0}, upload=480458]: Uploading piece... Dec 28 11:25:35.932729840 (547h56m35.803812875s) [upload=480458]: Next returned nil segment Dec 28 11:25:35.932749927 (547h56m35.803832962s) [upload=480458]: Waiting for error group managing segments... Dec 28 11:25:36.263262379 (547h56m36.134345424s) [noise=true , address=193.149.164.143:4695 , piece_id=XQVRYNRGL6TK6QJ6UUH6DYCLUZ5JDXRC4TTBWZ3I3BMRSXBMFA5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.490162414 (547h56m36.361245449s) [noise=true , address=145.79.227.33:4878 , piece_id=YB7TH7MUPPMTXSD3TXH55Y2BS3JLZ5YT35SKLKK72BGRZOO6QMUQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.491849427 (547h56m36.362932462s) [noise=true , address=145.79.227.43:4174 , piece_id=AKMC6XVLDVSFXSMX6EBKM2BULGQYS4PUYR6MNUQXPGFV43I73YFA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.523702000 (547h56m36.394785045s) [noise=true , address=145.79.227.11:4745 , piece_id=VDOGZCCCSGHNURRMJ74WGGFBNJC6PQCFZQFQ57DW7TL7U3NI5GNQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.539014337 (547h56m36.410097373s) [noise=true , address=145.79.227.46:4613 , piece_id=PKGBMUVJ7EDLOEFECMFKWR45HGBD6H5COQLBZT2GUZBZG4OOXVUA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.703251933 (547h56m36.574334978s) [noise=true , address=77.90.31.10:4533 , piece_id=EJ4BXYL3VHGJDQSWTRBK2MDIMBYZWVGPNGVBAW5CXATDNIG3SO3A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.774268671 (547h56m36.645351706s) [noise=true , address=77.90.31.14:4674 , piece_id=BI5O2ZQMR4SU4DMWS5MAITBOG32KF7QVAZS5WYN45M6LGRXQTDRA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.792078694 (547h56m36.663161729s) [noise=true , address=77.90.31.19:4869 , piece_id=YGORBKU5FW7EQW2IT2754N52B5LZCSBINVZN6BH65TM7UUBOEI5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.821706180 (547h56m36.692789215s) [noise=true , address=45.91.116.10:4096 , piece_id=ITZILOHFHA4COSK5RQZPDVSUK3AHSVOKEJPTRUV53KYLLQHMLICA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:36.880056288 (547h56m36.751139323s) [noise=true , address=77.90.31.15:4708 , piece_id=WI5BBWEG53LKOJIV6PVHKWIRTKURJPVZXNCSGQYEA3FERSIEZ7UA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.006605499 (547h56m36.877688534s) [noise=true , address=77.90.31.13:4660 , piece_id=4JYN22KDY7PEM6UT3RGTMRGHADRRMF4ICKWCI66ZSJIUSGJB73DA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.373015596 (547h56m37.244098631s) [noise=true , address=145.79.227.40:4077 , piece_id=XZADB7VPNQBZE6P6XZDPGCRGTLEDJP7ZQHSTXEE7DCMOUGZ4QO5A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.449754611 (547h56m37.320837646s) [noise=true , address=145.79.227.44:4204 , piece_id=RZXLRWR7H2YOYPO2HF476QQNIDGLKCWESRNLOJGUCMWTM3UDIQLA, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.484055044 (547h56m37.355138079s) [noise=true , address=145.79.227.30:4790 , piece_id=AJJSRQJJQQBQZQYKKGJXB3AZMBGNXM5CSVOPUGTX2QMPDRTDQG4A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: <nil> Dec 28 11:25:37.484076976 (547h56m37.355160021s) [seg_pos={0 0}, upload=480458]: Segment reached optimal threshold of 14 pieces. Dec 28 11:25:37.484306723 (547h56m37.355389788s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484316702 (547h56m37.355399737s) [noise=true , address=77.90.31.21:4893 , piece_id=HXJCPWAMNWQWKTS4G4E6MYYEUF6HIBNDC2OT3E7WR3FEF54NKE6Q, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484408347 (547h56m37.355491382s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484421281 (547h56m37.355504346s) [noise=true , address=45.91.116.7:4013 , piece_id=KEE6RJG4QIGBO7A4ESGGUI27ZJTWJSMAIJVO52P4MGM6H7WLBW7A, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484463993 (547h56m37.355547028s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35946->64.49.13.169:4705: use of closed network connection Dec 28 11:25:37.484472289 (547h56m37.355555334s) [noise=true , address=64.49.13.169:4705 , piece_id=OUK44JCG5HOWYP5P6WXE4VF4SXX6L5ZU7JBI7UHB6S76BZAEB3IQ, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484625149 (547h56m37.355708194s) [noise=true , address=145.79.227.49:4364 , piece_id=BU3SILKDTQ5GDMER3UFVQXELR4TTO7NTOFF3AHEBOVNYWFSLWINQ, seg_pos={0 0}, upload=480458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 28 11:25:37.484632975 (547h56m37.35571601s) [noise=true , address=145.79.227.49:4364 , piece_id=BU3SILKDTQ5GDMER3UFVQXELR4TTO7NTOFF3AHEBOVNYWFSLWINQ, seg_pos={0 0}, upload=480458]: Long tail cancellation due to optimal threshold Dec 28 11:25:37.484725450 (547h56m37.355808496s) [seg_pos={0 0}, upload=480458]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:25:37.484809191 (547h56m37.355892226s) [upload=480458]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:25:37.484815603 (547h56m37.355898638s) [upload=480458]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:25:37.774194368 (547h56m37.645277413s) [upload=480458]: Done uploading segments... Dec 28 11:25:37.774217130 (547h56m37.645300176s) [upload=480458]: Upload finished. err: <nil> Duration: 1m0.00331467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X...........] duration: 5.78771496s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:26:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.787731942s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.] duration: 3.47839993s trace: https://jaeger.dev.storj.io/trace/3abb0f5e27a2d8c FAIL (2025-12-28 11:26:09 UTC) orbiter warning: context deadline exceeded Duration: 3.478422613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X..........] duration: 5.85835148s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:26:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.858391417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X.........] duration: 7.570169427s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:25:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 7.570195907s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X] duration: 5.000775797s trace: https://jaeger.dev.storj.io/trace/3abb0f5e27a2d8c FAIL (2025-12-28 11:25:43 UTC) orbiter warning: context deadline exceeded Duration: 5.000794193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X........X...........] duration: 37.689958511s trace: https://jaeger.dev.storj.io/trace/48bdb131d72f6dbe FAIL (2025-12-28 11:25:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855848051 (547h44m22.726931086s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855874902 (547h44m22.726957947s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856273391 (547h44m22.727356426s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856301254 (547h44m22.727384299s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856685706 (547h44m22.727768751s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857102991 (547h44m22.728186046s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857216196 (547h44m22.728299241s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857564579 (547h44m22.728647614s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857862176 (547h44m22.728945211s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858110980 (547h44m22.729194025s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858203376 (547h44m22.729286421s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858702256 (547h44m22.729785301s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859235351 (547h44m22.730318386s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859826286 (547h44m22.730909321s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859890889 (547h44m22.730973924s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:24.219555199 (547h44m24.090638234s) [upload=480284]: Next returned nil segment Dec 28 11:13:24.219566792 (547h44m24.090649827s) [upload=480284]: Waiting for error group managing segments... Dec 28 11:13:24.575670854 (547h44m24.446753889s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.717265714 (547h44m24.588348749s) [noise=true , address=193.149.164.143:4695 , piece_id=3ACSK6DQ2BPOVBTN3JDRBTNY27G2TSWPBJZEVMX4UKARWCVTQI5Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.743823363 (547h44m24.614906409s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.746223524 (547h44m24.617306569s) [noise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.747489475 (547h44m24.61857252s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.750174588 (547h44m24.621257623s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.912872813 (547h44m24.783955848s) [noise=true , address=77.90.31.11:4577 , piece_id=2XQKV336Q24H6NRZ6PH53ROBEAMMXKR423ZGOEKLKDBIEUVOGG3A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.983865104 (547h44m24.854948139s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.046596836 (547h44m24.917679871s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.050882637 (547h44m24.921965682s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.142228164 (547h44m25.013311209s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.631524266 (547h44m25.502607302s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.657378687 (547h44m25.528461732s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659003652 (547h44m25.530086707s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659019602 (547h44m25.530102717s) [seg_pos={0 0}, upload=480284]: Segment reached optimal threshold of 14 pieces. Dec 28 11:13:25.659239089 (547h44m25.530322134s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659248087 (547h44m25.530331132s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659273324 (547h44m25.530356359s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659282613 (547h44m25.530365658s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659460782 (547h44m25.530543817s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection Dec 28 11:13:25.659467174 (547h44m25.530550209s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659680961 (547h44m25.530764026s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659692282 (547h44m25.530775327s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659716248 (547h44m25.530799353s) [seg_pos={0 0}, upload=480284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:13:25.659827760 (547h44m25.530910805s) [upload=480284]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:13:25.659834354 (547h44m25.530917399s) [upload=480284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:13:25.949930415 (547h44m25.82101345s) [upload=480284]: Done uploading segments... Dec 28 11:13:25.949958028 (547h44m25.821041063s) [upload=480284]: Upload finished. err: <nil> Duration: 37.689973008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X........] duration: 5.893131734s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:25:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.893169255s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX..] duration: 3.437987688s trace: https://jaeger.dev.storj.io/trace/15359ac314b9c228 FAIL (2025-12-28 11:24:36 UTC) orbiter warning: context deadline exceeded Duration: 3.438012915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X........X..........] duration: 40.239856016s trace: https://jaeger.dev.storj.io/trace/48bdb131d72f6dbe FAIL (2025-12-28 11:24:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855848051 (547h44m22.726931086s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855874902 (547h44m22.726957947s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856273391 (547h44m22.727356426s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856301254 (547h44m22.727384299s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856685706 (547h44m22.727768751s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857102991 (547h44m22.728186046s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857216196 (547h44m22.728299241s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857564579 (547h44m22.728647614s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857862176 (547h44m22.728945211s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858110980 (547h44m22.729194025s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858203376 (547h44m22.729286421s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858702256 (547h44m22.729785301s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859235351 (547h44m22.730318386s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859826286 (547h44m22.730909321s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859890889 (547h44m22.730973924s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:24.219555199 (547h44m24.090638234s) [upload=480284]: Next returned nil segment Dec 28 11:13:24.219566792 (547h44m24.090649827s) [upload=480284]: Waiting for error group managing segments... Dec 28 11:13:24.575670854 (547h44m24.446753889s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.717265714 (547h44m24.588348749s) [noise=true , address=193.149.164.143:4695 , piece_id=3ACSK6DQ2BPOVBTN3JDRBTNY27G2TSWPBJZEVMX4UKARWCVTQI5Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.743823363 (547h44m24.614906409s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.746223524 (547h44m24.617306569s) [noise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.747489475 (547h44m24.61857252s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.750174588 (547h44m24.621257623s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.912872813 (547h44m24.783955848s) [noise=true , address=77.90.31.11:4577 , piece_id=2XQKV336Q24H6NRZ6PH53ROBEAMMXKR423ZGOEKLKDBIEUVOGG3A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.983865104 (547h44m24.854948139s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.046596836 (547h44m24.917679871s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.050882637 (547h44m24.921965682s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.142228164 (547h44m25.013311209s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.631524266 (547h44m25.502607302s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.657378687 (547h44m25.528461732s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659003652 (547h44m25.530086707s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659019602 (547h44m25.530102717s) [seg_pos={0 0}, upload=480284]: Segment reached optimal threshold of 14 pieces. Dec 28 11:13:25.659239089 (547h44m25.530322134s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659248087 (547h44m25.530331132s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659273324 (547h44m25.530356359s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659282613 (547h44m25.530365658s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659460782 (547h44m25.530543817s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection Dec 28 11:13:25.659467174 (547h44m25.530550209s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659680961 (547h44m25.530764026s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659692282 (547h44m25.530775327s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659716248 (547h44m25.530799353s) [seg_pos={0 0}, upload=480284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:13:25.659827760 (547h44m25.530910805s) [upload=480284]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:13:25.659834354 (547h44m25.530917399s) [upload=480284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:13:25.949930415 (547h44m25.82101345s) [upload=480284]: Done uploading segments... Dec 28 11:13:25.949958028 (547h44m25.821041063s) [upload=480284]: Upload finished. err: <nil> Duration: 40.239886624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X.......] duration: 6.021102592s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.021131417s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX.] duration: 5.000839149s trace: https://jaeger.dev.storj.io/trace/15359ac314b9c228 FAIL (2025-12-28 11:23:58 UTC) orbiter warning: context deadline exceeded Duration: 5.000884575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X...XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXXX] duration: 5.00101306s trace: https://jaeger.dev.storj.io/trace/15359ac314b9c228 FAIL (2025-12-28 11:23:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001034762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...........XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X......] duration: 5.75143545s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:23:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.751457362s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X...XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XXX] duration: 5.001275954s trace: https://jaeger.dev.storj.io/trace/19361deb63bed1b0 FAIL (2025-12-28 11:23:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001318215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration: 35.623446478s trace: https://jaeger.dev.storj.io/trace/48bdb131d72f6dbe FAIL (2025-12-28 11:23:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855848051 (547h44m22.726931086s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855874902 (547h44m22.726957947s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856273391 (547h44m22.727356426s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856301254 (547h44m22.727384299s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856685706 (547h44m22.727768751s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857102991 (547h44m22.728186046s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857216196 (547h44m22.728299241s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857564579 (547h44m22.728647614s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857862176 (547h44m22.728945211s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858110980 (547h44m22.729194025s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858203376 (547h44m22.729286421s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858702256 (547h44m22.729785301s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859235351 (547h44m22.730318386s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859826286 (547h44m22.730909321s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859890889 (547h44m22.730973924s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:24.219555199 (547h44m24.090638234s) [upload=480284]: Next returned nil segment Dec 28 11:13:24.219566792 (547h44m24.090649827s) [upload=480284]: Waiting for error group managing segments... Dec 28 11:13:24.575670854 (547h44m24.446753889s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.717265714 (547h44m24.588348749s) [noise=true , address=193.149.164.143:4695 , piece_id=3ACSK6DQ2BPOVBTN3JDRBTNY27G2TSWPBJZEVMX4UKARWCVTQI5Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.743823363 (547h44m24.614906409s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.746223524 (547h44m24.617306569s) [noise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.747489475 (547h44m24.61857252s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.750174588 (547h44m24.621257623s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.912872813 (547h44m24.783955848s) [noise=true , address=77.90.31.11:4577 , piece_id=2XQKV336Q24H6NRZ6PH53ROBEAMMXKR423ZGOEKLKDBIEUVOGG3A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.983865104 (547h44m24.854948139s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.046596836 (547h44m24.917679871s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.050882637 (547h44m24.921965682s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.142228164 (547h44m25.013311209s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.631524266 (547h44m25.502607302s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.657378687 (547h44m25.528461732s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659003652 (547h44m25.530086707s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659019602 (547h44m25.530102717s) [seg_pos={0 0}, upload=480284]: Segment reached optimal threshold of 14 pieces. Dec 28 11:13:25.659239089 (547h44m25.530322134s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659248087 (547h44m25.530331132s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659273324 (547h44m25.530356359s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659282613 (547h44m25.530365658s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659460782 (547h44m25.530543817s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection Dec 28 11:13:25.659467174 (547h44m25.530550209s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659680961 (547h44m25.530764026s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659692282 (547h44m25.530775327s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659716248 (547h44m25.530799353s) [seg_pos={0 0}, upload=480284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:13:25.659827760 (547h44m25.530910805s) [upload=480284]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:13:25.659834354 (547h44m25.530917399s) [upload=480284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:13:25.949930415 (547h44m25.82101345s) [upload=480284]: Done uploading segments... Dec 28 11:13:25.949958028 (547h44m25.821041063s) [upload=480284]: Upload finished. err: <nil> Duration: 35.623472196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...........XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X.....] duration: 5.961686353s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:22:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 5.961715297s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..X...XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..XX] duration: 5.00162158s trace: https://jaeger.dev.storj.io/trace/47df7300a2dfd249 FAIL (2025-12-28 11:22:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001645345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X........X........] duration: 26.325980103s trace: https://jaeger.dev.storj.io/trace/48bdb131d72f6dbe FAIL (2025-12-28 11:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855848051 (547h44m22.726931086s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.855874902 (547h44m22.726957947s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856273391 (547h44m22.727356426s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856301254 (547h44m22.727384299s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.856685706 (547h44m22.727768751s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857102991 (547h44m22.728186046s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857216196 (547h44m22.728299241s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857564579 (547h44m22.728647614s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.857862176 (547h44m22.728945211s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858110980 (547h44m22.729194025s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858203376 (547h44m22.729286421s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.858702256 (547h44m22.729785301s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859235351 (547h44m22.730318386s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859826286 (547h44m22.730909321s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:22.859890889 (547h44m22.730973924s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Uploading piece... Dec 28 11:13:24.219555199 (547h44m24.090638234s) [upload=480284]: Next returned nil segment Dec 28 11:13:24.219566792 (547h44m24.090649827s) [upload=480284]: Waiting for error group managing segments... Dec 28 11:13:24.575670854 (547h44m24.446753889s) [noise=true , address=64.49.13.140:4516 , piece_id=5ESZJUJNL5MG4U7RVSOCHXQHL5FDD5IZATLI3HMIVN7XXKFE5KDA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.717265714 (547h44m24.588348749s) [noise=true , address=193.149.164.143:4695 , piece_id=3ACSK6DQ2BPOVBTN3JDRBTNY27G2TSWPBJZEVMX4UKARWCVTQI5Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.743823363 (547h44m24.614906409s) [noise=true , address=145.79.227.29:4958 , piece_id=R27HRPXBPYT2RRS3DYVUHAY3FXV73YRSZORJ5PHKDNWYAPZFOBPA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.746223524 (547h44m24.617306569s) [noise=true , address=145.79.227.46:4614 , piece_id=D6IRE4PW7MDO6NAV5P3TL4EPPCJXR45KCIHSGFV4F6LTYPP337NQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.747489475 (547h44m24.61857252s) [noise=true , address=145.79.227.43:4160 , piece_id=2S2NLD3EVEIG3RKZGK6SJOJ2DNBLEBBS6RFQ73R43ERORPHNHOHQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.750174588 (547h44m24.621257623s) [noise=true , address=145.79.227.52:4449 , piece_id=OPLGX3WYFOJHNHMN7BDKXM5INWXF4AXHC3BRIDEYJ2BJA63UAZXQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.912872813 (547h44m24.783955848s) [noise=true , address=77.90.31.11:4577 , piece_id=2XQKV336Q24H6NRZ6PH53ROBEAMMXKR423ZGOEKLKDBIEUVOGG3A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:24.983865104 (547h44m24.854948139s) [noise=true , address=77.90.31.17:4776 , piece_id=F3WQAAIHDW3XIJZO3ULP3X665ANFG2IBSAGJUA6ZWCH2U2L44LFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.046596836 (547h44m24.917679871s) [noise=true , address=45.91.116.4:4943 , piece_id=MHATE3SHULK2XBM7ZCY5PWDEY6ZJKZZGWL4MR36OPZAIENZ7342A, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.050882637 (547h44m24.921965682s) [noise=true , address=45.91.116.13:4187 , piece_id=4NB7SZTQH43EVCB4ET6MRIYPVK5Q6BXAEXVA4HYEZDU6WCDGNEPQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.142228164 (547h44m25.013311209s) [noise=true , address=77.90.31.12:4606 , piece_id=STJRY3VBBK6CCNTOQPLTZXILPU6W52ENWATWITXXN44EU4YKYFGQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.631524266 (547h44m25.502607302s) [noise=true , address=145.79.227.30:4796 , piece_id=76USZLRAEQLILHN56M2XB5J3PV5ZDB2HUFLMICWPA5RETVJTOKTA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.657378687 (547h44m25.528461732s) [noise=true , address=145.79.227.44:4213 , piece_id=H6BL2YYJSJDQYFICA7ITRKKI75M3ETPW7S4YZMDVDNQ5LN7GAVNQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659003652 (547h44m25.530086707s) [noise=true , address=145.79.227.34:4937 , piece_id=GYQAYHYMEFVULYYGBGXDCM3I4MSDLCQQOUSRO3PYWWOLJVKGJZJQ, seg_pos={0 0}, upload=480284]: Done uploading piece. err: <nil> Dec 28 11:13:25.659019602 (547h44m25.530102717s) [seg_pos={0 0}, upload=480284]: Segment reached optimal threshold of 14 pieces. Dec 28 11:13:25.659239089 (547h44m25.530322134s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659248087 (547h44m25.530331132s) [noise=true , address=77.90.31.16:4745 , piece_id=6H7AUN6FZ3GOQDHJQ32XVIRWCF62VCIQXDOIRNTMF7OP5VRZHOEA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659273324 (547h44m25.530356359s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659282613 (547h44m25.530365658s) [noise=true , address=145.79.227.40:4076 , piece_id=ONMN5TXAG5IOHL4WOKUIIJADRMUFRUGK3AXJCVMVE7AVWOVVHT6Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659460782 (547h44m25.530543817s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36090->64.49.13.169:4707: use of closed network connection Dec 28 11:13:25.659467174 (547h44m25.530550209s) [noise=true , address=64.49.13.169:4707 , piece_id=QBCPEUCF2BU6265AZWFT2XFKBXAILVWGN5MXRSPQM3ECY637KIFA, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659680961 (547h44m25.530764026s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 11:13:25.659692282 (547h44m25.530775327s) [noise=true , address=77.90.31.18:4819 , piece_id=BLMFQVKHTCTBRZYNF5TRHQMWMV4Y32KOEBX276GAM6UCTN3L3F7Q, seg_pos={0 0}, upload=480284]: Long tail cancellation due to optimal threshold Dec 28 11:13:25.659716248 (547h44m25.530799353s) [seg_pos={0 0}, upload=480284]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:13:25.659827760 (547h44m25.530910805s) [upload=480284]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:13:25.659834354 (547h44m25.530917399s) [upload=480284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:13:25.949930415 (547h44m25.82101345s) [upload=480284]: Done uploading segments... Dec 28 11:13:25.949958028 (547h44m25.821041063s) [upload=480284]: Upload finished. err: <nil> Duration: 26.326006283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX...........XXX.X.X.X...X...XXX...X.XX.X..XX..X.......X....] duration: 6.132454337s trace: https://jaeger.dev.storj.io/trace/7cddee23e3308a97 FAIL (2025-12-28 11:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ) context deadline exceeded ======LOGS====== 1:17:52.332923746 (547h48m52.204006781s) [upload=480346]: Starting upload Dec 28 11:17:52.333009510 (547h48m52.204092545s) [upload=480346]: Done starting upload Dec 28 11:17:52.333131422 (547h48m52.204214457s) [upload=480346]: Uploading segments... Dec 28 11:17:52.333306695 (547h48m52.204389741s) [upload=480346]: Got next segment. Inline: false Dec 28 11:17:52.333355148 (547h48m52.204438203s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:17:52.333361059 (547h48m52.204444115s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:17:52.581966708 (547h48m52.453049753s) [seg_pos={0 0}, upload=480346]: Begin upload segment... Dec 28 11:17:52.582035629 (547h48m52.453118664s) [seg_pos={0 0}, upload=480346]: Done begin upload segment. Dec 28 11:17:52.582087437 (547h48m52.453170472s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582136501 (547h48m52.453219546s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.582748286 (547h48m52.453831321s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583155491 (547h48m52.454238536s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583509034 (547h48m52.454592069s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.583863239 (547h48m52.454946274s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584255877 (547h48m52.455338922s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584676889 (547h48m52.455759934s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.584684052 (547h48m52.455767097s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585137425 (547h48m52.45622047s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585184925 (547h48m52.456267961s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.585579407 (547h48m52.456662442s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586035546 (547h48m52.457118591s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586488618 (547h48m52.457571664s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.586893429 (547h48m52.457976474s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:52.587285346 (547h48m52.458368391s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Uploading piece... Dec 28 11:17:54.302888403 (547h48m54.173971438s) [upload=480346]: Next returned nil segment Dec 28 11:17:54.302901778 (547h48m54.173984824s) [upload=480346]: Waiting for error group managing segments... Dec 28 11:17:54.583059666 (547h48m54.454142701s) [noise=true , address=145.79.227.15:4873 , piece_id=73OEJKKZQPEPE4UU3MTACKXRVL65DTMWWFEK76AB2OGAKV5XMGPQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594022602 (547h48m54.465105637s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMSHS2XBQN7MIMGSTO7FGLUAWFL4KDSXHKKQUBQGHPBIGVFSHNA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594547631 (547h48m54.465630667s) [noise=true , address=145.79.227.7:4295 , piece_id=SEYEIZVRTLHLSJCUPKYOJGTWW7BZ2KUXRZVASK6WM7ZWJJHNQ76Q, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.594730239 (547h48m54.465813284s) [noise=true , address=145.79.227.39:4014 , piece_id=YMWKSWDM6XZM2BWYNHA4E642O6RXQ6C2TIHJBFXWYVKHNW2LSHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.595947357 (547h48m54.467030392s) [noise=true , address=145.79.227.51:4410 , piece_id=BPPCKEXGDEQMNMOBWY5RRVCEPDXNFRQS4YNBOER2VV2AUZR5IJIA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.597694323 (547h48m54.468777358s) [noise=true , address=145.79.227.4:4187 , piece_id=OGCJQ3QYVU4L4CICDBEM3ARB4KLWCMO5IUBWYMLNRCSBSBJ62UGA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.600744541 (547h48m54.471827587s) [noise=true , address=145.79.227.13:4803 , piece_id=64W6CMVX5N5PJHD5XOGSFLVDOCVWUMDF4AP6WU6C2BYSUTC3BNOA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.601742722 (547h48m54.472825758s) [noise=true , address=145.79.227.45:4231 , piece_id=GW3IUPGXVWATNPBU7MNFJTHY6BDKG6KSZ5JYBM7E5X5WPHVOEHPA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.609542895 (547h48m54.48062594s) [noise=true , address=145.79.227.6:4259 , piece_id=I4OJ5IGGYMM36SB6UPJXFCBFFKHTQF77V6U3R464E2ZGRCIUHVDQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.613290060 (547h48m54.484373115s) [noise=true , address=145.79.227.16:4909 , piece_id=MK7C6C6N3EPESCFJG2D6FYV3VZY5ZQ6YDE3ZTCO7XXLSUZSWV4NA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.755778162 (547h48m54.626861207s) [noise=true , address=77.90.31.10:4517 , piece_id=SONBUGHK4KQBY5KRRVXTDBMFYYHHDCEBAKIC3F3LTAV7GO4O5PJQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.772472341 (547h48m54.643555376s) [noise=true , address=77.90.31.13:4629 , piece_id=FXP76PCE2YDSZ55BPIDE2DOMWPF6UNXVJJHHTN3SKDFFFJPA3WMA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842877704 (547h48m54.713960749s) [noise=true , address=45.91.116.12:4226 , piece_id=SCEVNUDUPNA3AFETSN2BVOOVFN5AWUYBJR2MEOWA4PU7IJIVFGHQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: <nil> Dec 28 11:17:54.842905848 (547h48m54.713988893s) [seg_pos={0 0}, upload=480346]: Segment reached optimal threshold of 13 pieces. Dec 28 11:17:54.843187033 (547h48m54.714270078s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843195679 (547h48m54.714278724s) [noise=true , address=45.91.116.6:4973 , piece_id=AWUFGHR5F6TW44NRM24DY4V6HQB5XMOQDNXKNJ5J3SPJ2ABYOQFA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843530536 (547h48m54.714613602s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 11:17:54.843541789 (547h48m54.714624834s) [noise=true , address=77.90.31.12:4590 , piece_id=HSSQ7HXPJNRS4NYG3RL6EZ6DH37ZPMPJ62IWPTUPYRVFU4PRGXQQ, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843553561 (547h48m54.714636606s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35218->45.91.116.10:4084: use of closed network connection Dec 28 11:17:54.843563860 (547h48m54.714646905s) [noise=true , address=45.91.116.10:4084 , piece_id=EPHBH3VJRWWHD5DMQOLSWZWFOMLVIV67SFBZK6L2WS76QGDU6WDA, seg_pos={0 0}, upload=480346]: Long tail cancellation due to optimal threshold Dec 28 11:17:54.843586714 (547h48m54.714669749s) [seg_pos={0 0}, upload=480346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:17:54.843670584 (547h48m54.714753619s) [upload=480346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:17:54.843677868 (547h48m54.714760913s) [upload=480346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:17:55.134037951 (547h48m55.005120986s) [upload=480346]: Done uploading segments... Dec 28 11:17:55.134058361 (547h48m55.005141406s) [upload=480346]: Upload finished. err: <nil> Duration: 6.13248221s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...X...XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..X] duration: 5.001858231s trace: https://jaeger.dev.storj.io/trace/28fb0301a84a4484 FAIL (2025-12-28 11:21:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001882818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....X...XX......XXXX.....X.X...X.....XX.XX..XXX...X.X....X..] duration: 3.400987482s trace: https://jaeger.dev.storj.io/trace/a36b84e38a12b0 FAIL (2025-12-28 11:21:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.401033109s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)