Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.........X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX....X.X] duration: 5.001601184s trace: https://jaeger.dev.storj.io/trace/5275d8e37dcb808d FAIL (2025-12-27 13:30:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001618537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.............................X...X...X..........X] duration: 1m0.003042295s trace: https://jaeger.dev.storj.io/trace/505f3754ac75670e FAIL (2025-12-27 13:29:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, Piece ID: 2LQMPKIOCNPIZXVRBEMG7VKSIS4NHUTGFROT6SRR62PAP5SDZM6Q) context deadline exceeded; piecestore: (Node ID: 1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, Piece ID: KQUGJP3SWGVMAUJURINAQJXYW77M3F6RBJCZAR7UYR7TTKHIXPLA) context deadline exceeded; piecestore: (Node ID: 1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, Piece ID: E5S5XONCPAZY77L7LVWAJX3M5Q23Z2MBNO7W6453SGRFJBPDKSRQ) context deadline exceeded; piecestore: (Node ID: 12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, Piece ID: KB2ZPPHYRAG57FDZHE2JGKKOPDJ3Y4LX7K4QQQNMDUXNKGKDSIQQ) context deadline exceeded ======LOGS====== .43:4187 , piece_id=KB2ZPPHYRAG57FDZHE2JGKKOPDJ3Y4LX7K4QQQNMDUXNKGKDSIQQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.895592615 (525h59m51.76667566s) [noise=true , address=64.49.13.139:4469 , piece_id=5EABHD5B3W756QMYIAQG72OIOOICVBWMPFVQGLPWZWMCFUV377NA, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.896143324 (525h59m51.767226369s) [noise=true , address=145.79.227.46:4622 , piece_id=4KQOF4Q7WL6O7NNULLBIZRX7MJ4OXB6IOAS4U3SWZV55F7BB6DLQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.896170045 (525h59m51.76725308s) [noise=true , address=77.90.31.15:4723 , piece_id=KQUGJP3SWGVMAUJURINAQJXYW77M3F6RBJCZAR7UYR7TTKHIXPLA, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.896656181 (525h59m51.767739226s) [noise=true , address=77.90.31.14:4668 , piece_id=E5S5XONCPAZY77L7LVWAJX3M5Q23Z2MBNO7W6453SGRFJBPDKSRQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.896939871 (525h59m51.768022916s) [noise=true , address=145.79.227.7:4305 , piece_id=EWAS4VXPGPRNZZQ6RP2PHMO7LTR6X2X4QI4UHSUZD2DEZ3LLKFBQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.897132758 (525h59m51.768215794s) [noise=true , address=64.49.13.169:4706 , piece_id=HCDMFH5OTSXBWKLKK2YFNPZRLTI4DMO6JANSYJXL2WVGV74MY3EQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.897594217 (525h59m51.768677252s) [noise=true , address=77.90.31.10:4530 , piece_id=YWYLY7MMUR4ILHALLVDH4B67C63GZ6NOY5OXIYC3LI6AQITGWNJQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.897649573 (525h59m51.768732618s) [noise=true , address=145.79.227.30:4782 , piece_id=AHC5VWU7KANRVBG64JQKWCBQ2BXTAMZVGXMFLJ5R4FSIMZNZGNHQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.898209819 (525h59m51.769292865s) [noise=true , address=145.79.227.33:4892 , piece_id=CXRUXHYSLEJTYXATD5BITJ3K33ZJOSFTE6OLX36BKRE2TTA4N54Q, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.898314068 (525h59m51.769397113s) [noise=true , address=45.91.116.13:4179 , piece_id=E2EKVWD37BF7KFE64NT55J54G5O45RISDMXRPPB44GIEAWFRUPHA, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.898735590 (525h59m51.769818626s) [noise=true , address=77.90.31.21:4903 , piece_id=2LQMPKIOCNPIZXVRBEMG7VKSIS4NHUTGFROT6SRR62PAP5SDZM6Q, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.898798611 (525h59m51.769881656s) [noise=true , address=193.149.164.143:4697 , piece_id=THGFEMSKEJDPT45MJQL76ZDFDJTWXOB3TZCXTQGQI77IMTIFG2UA, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.899314663 (525h59m51.770397698s) [noise=true , address=77.90.31.18:4815 , piece_id=JLG3FJLWHT752U5D2JYGEOIJYTFNTMZEA5GMAQYFTCL6GYOLT7KQ, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:51.903308868 (525h59m51.774391913s) [noise=true , address=145.79.227.34:4913 , piece_id=AU3HIR2ZBEX3SA7TJZLK6HPXKUL6MFJE5YRD2SUMYBG4JUPFVFXA, seg_pos={0 0}, upload=461387]: Uploading piece... Dec 27 13:28:53.206016181 (525h59m53.077099217s) [upload=461387]: Next returned nil segment Dec 27 13:28:53.206105391 (525h59m53.077188436s) [upload=461387]: Waiting for error group managing segments... Dec 27 13:28:53.536403005 (525h59m53.40748605s) [noise=true , address=64.49.13.139:4469 , piece_id=5EABHD5B3W756QMYIAQG72OIOOICVBWMPFVQGLPWZWMCFUV377NA, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:53.759696184 (525h59m53.630779219s) [noise=true , address=145.79.227.46:4622 , piece_id=4KQOF4Q7WL6O7NNULLBIZRX7MJ4OXB6IOAS4U3SWZV55F7BB6DLQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:53.770854342 (525h59m53.641937377s) [noise=true , address=145.79.227.7:4305 , piece_id=EWAS4VXPGPRNZZQ6RP2PHMO7LTR6X2X4QI4UHSUZD2DEZ3LLKFBQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:53.855181235 (525h59m53.72626428s) [noise=true , address=145.79.227.43:4187 , piece_id=KB2ZPPHYRAG57FDZHE2JGKKOPDJ3Y4LX7K4QQQNMDUXNKGKDSIQQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:53.891962603 (525h59m53.763045638s) [noise=true , address=145.79.227.34:4913 , piece_id=AU3HIR2ZBEX3SA7TJZLK6HPXKUL6MFJE5YRD2SUMYBG4JUPFVFXA, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.170927652 (525h59m54.042010697s) [noise=true , address=45.91.116.13:4179 , piece_id=E2EKVWD37BF7KFE64NT55J54G5O45RISDMXRPPB44GIEAWFRUPHA, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.208435443 (525h59m54.079518478s) [noise=true , address=45.91.116.8:4066 , piece_id=KHOZA2WOPBL4ZCDXEWDBNBJZYJCMCUD4X6FNA6K3J7FUTVF3VTDQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.284487519 (525h59m54.155570554s) [noise=true , address=77.90.31.15:4723 , piece_id=KQUGJP3SWGVMAUJURINAQJXYW77M3F6RBJCZAR7UYR7TTKHIXPLA, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.468576497 (525h59m54.339659532s) [noise=true , address=193.149.164.143:4697 , piece_id=THGFEMSKEJDPT45MJQL76ZDFDJTWXOB3TZCXTQGQI77IMTIFG2UA, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.501690602 (525h59m54.372773647s) [noise=true , address=77.90.31.10:4530 , piece_id=YWYLY7MMUR4ILHALLVDH4B67C63GZ6NOY5OXIYC3LI6AQITGWNJQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.708771840 (525h59m54.579854875s) [noise=true , address=145.79.227.30:4782 , piece_id=AHC5VWU7KANRVBG64JQKWCBQ2BXTAMZVGXMFLJ5R4FSIMZNZGNHQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.721228310 (525h59m54.592311345s) [noise=true , address=145.79.227.44:4219 , piece_id=6HCI25MJYKAP32RW6UG537AZH3GCECZ6NLWDC2UOAM6VFLIODSMQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.746750959 (525h59m54.617834004s) [noise=true , address=77.90.31.14:4668 , piece_id=E5S5XONCPAZY77L7LVWAJX3M5Q23Z2MBNO7W6453SGRFJBPDKSRQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.827637321 (525h59m54.698720356s) [noise=true , address=77.90.31.21:4903 , piece_id=2LQMPKIOCNPIZXVRBEMG7VKSIS4NHUTGFROT6SRR62PAP5SDZM6Q, seg_pos={0 0}, upload=461387]: Done uploading piece. err: <nil> Dec 27 13:28:54.827664643 (525h59m54.698747688s) [seg_pos={0 0}, upload=461387]: Segment reached optimal threshold of 14 pieces. Dec 27 13:28:54.827869453 (525h59m54.698952499s) [noise=true , address=145.79.227.40:4058 , piece_id=ROYMXBWYDQ32OFDIPXTPEIFACSQUSHS6DHL5HA37OAH74IWQKK5Q, seg_pos={0 0}, upload=461387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 27 13:28:54.827875335 (525h59m54.69895837s) [noise=true , address=145.79.227.40:4058 , piece_id=ROYMXBWYDQ32OFDIPXTPEIFACSQUSHS6DHL5HA37OAH74IWQKK5Q, seg_pos={0 0}, upload=461387]: Long tail cancellation due to optimal threshold Dec 27 13:28:54.827989492 (525h59m54.699072527s) [noise=true , address=145.79.227.33:4892 , piece_id=CXRUXHYSLEJTYXATD5BITJ3K33ZJOSFTE6OLX36BKRE2TTA4N54Q, seg_pos={0 0}, upload=461387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5): protocol: expected piece hash; context canceled; EOF Dec 27 13:28:54.827994521 (525h59m54.699077556s) [noise=true , address=145.79.227.33:4892 , piece_id=CXRUXHYSLEJTYXATD5BITJ3K33ZJOSFTE6OLX36BKRE2TTA4N54Q, seg_pos={0 0}, upload=461387]: Long tail cancellation due to optimal threshold Dec 27 13:28:54.828054025 (525h59m54.69913708s) [noise=true , address=64.49.13.169:4706 , piece_id=HCDMFH5OTSXBWKLKK2YFNPZRLTI4DMO6JANSYJXL2WVGV74MY3EQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35294->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35294->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35294->64.49.13.169:4706: use of closed network connection Dec 27 13:28:54.828062571 (525h59m54.699145616s) [noise=true , address=64.49.13.169:4706 , piece_id=HCDMFH5OTSXBWKLKK2YFNPZRLTI4DMO6JANSYJXL2WVGV74MY3EQ, seg_pos={0 0}, upload=461387]: Long tail cancellation due to optimal threshold Dec 27 13:28:54.828102256 (525h59m54.699185291s) [noise=true , address=77.90.31.18:4815 , piece_id=JLG3FJLWHT752U5D2JYGEOIJYTFNTMZEA5GMAQYFTCL6GYOLT7KQ, seg_pos={0 0}, upload=461387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: write tcp 188.166.225.170:45938->77.90.31.18:4815: use of closed network connection; write tcp 188.166.225.170:45938->77.90.31.18:4815: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45938->77.90.31.18:4815: use of closed network connection Dec 27 13:28:54.828107436 (525h59m54.699190471s) [noise=true , address=77.90.31.18:4815 , piece_id=JLG3FJLWHT752U5D2JYGEOIJYTFNTMZEA5GMAQYFTCL6GYOLT7KQ, seg_pos={0 0}, upload=461387]: Long tail cancellation due to optimal threshold Dec 27 13:28:54.828264034 (525h59m54.69934707s) [seg_pos={0 0}, upload=461387]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:28:54.828331703 (525h59m54.699414738s) [upload=461387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:28:54.828335831 (525h59m54.699418866s) [upload=461387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:28:55.127617726 (525h59m54.998700761s) [upload=461387]: Done uploading segments... Dec 27 13:28:55.127639989 (525h59m54.998723024s) [upload=461387]: Upload finished. err: <nil> Duration: 1m0.003057883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..........X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX....X.] duration: 5.001413455s trace: https://jaeger.dev.storj.io/trace/590ef8ad65c1b0d5 FAIL (2025-12-27 13:29:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001440777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X..XXX...........XX..X...XX......X..X...X.XXX....X...X...] duration: 6.459151304s trace: https://jaeger.dev.storj.io/trace/795eb7a89d3a1651 FAIL (2025-12-27 13:29:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ) context deadline exceeded ======LOGS====== Dec 27 13:27:17.774856677 (525h58m17.645939722s) [upload=461359]: Starting upload Dec 27 13:27:17.774949122 (525h58m17.646032198s) [upload=461359]: Done starting upload Dec 27 13:27:17.775049243 (525h58m17.646132308s) [upload=461359]: Uploading segments... Dec 27 13:27:17.775267238 (525h58m17.646350283s) [upload=461359]: Got next segment. Inline: false Dec 27 13:27:17.775307154 (525h58m17.646390199s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:27:17.775313998 (525h58m17.646397033s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:27:18.038975516 (525h58m17.910058561s) [seg_pos={0 0}, upload=461359]: Begin upload segment... Dec 27 13:27:18.039023948 (525h58m17.910106993s) [seg_pos={0 0}, upload=461359]: Done begin upload segment. Dec 27 13:27:18.039056890 (525h58m17.910139935s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039081648 (525h58m17.910164693s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039173783 (525h58m17.910256818s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039612218 (525h58m17.910695263s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039763686 (525h58m17.910846731s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040233381 (525h58m17.911316416s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040247377 (525h58m17.911330422s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040616851 (525h58m17.911699886s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040623193 (525h58m17.911706238s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041115771 (525h58m17.912198816s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041479203 (525h58m17.912562248s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041879445 (525h58m17.91296249s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041941533 (525h58m17.913024568s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042281711 (525h58m17.913364746s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042658880 (525h58m17.913741915s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042758278 (525h58m17.913841313s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:19.749190694 (525h58m19.620273729s) [upload=461359]: Next returned nil segment Dec 27 13:27:19.749207025 (525h58m19.62029006s) [upload=461359]: Waiting for error group managing segments... Dec 27 13:27:20.029249725 (525h58m19.90033276s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.039023607 (525h58m19.910106642s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.043960497 (525h58m19.915043543s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.054911700 (525h58m19.925994735s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.070576198 (525h58m19.941659253s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.073387070 (525h58m19.944470115s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.076402032 (525h58m19.947485067s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.337390148 (525h58m20.208473183s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.340016089 (525h58m20.211099124s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.376636711 (525h58m20.247719746s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.420938058 (525h58m20.292021094s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.468046200 (525h58m20.339129245s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939413045 (525h58m20.81049609s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939437562 (525h58m20.810520607s) [seg_pos={0 0}, upload=461359]: Segment reached optimal threshold of 13 pieces. Dec 27 13:27:20.939695693 (525h58m20.810778748s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939706114 (525h58m20.810789159s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939846972 (525h58m20.810930007s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939857632 (525h58m20.810940677s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939897418 (525h58m20.810980463s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939907246 (525h58m20.810990291s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939931553 (525h58m20.811014598s) [seg_pos={0 0}, upload=461359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:27:20.940016755 (525h58m20.811099801s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:27:20.940023999 (525h58m20.811107044s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:27:21.228670943 (525h58m21.099753978s) [upload=461359]: Done uploading segments... Dec 27 13:27:21.228692945 (525h58m21.09977598s) [upload=461359]: Upload finished. err: <nil> Duration: 6.459170871s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...........X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX....X] duration: 5.001513958s trace: https://jaeger.dev.storj.io/trace/590ef8ad65c1b0d5 FAIL (2025-12-27 13:28:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001530239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.............................X...X...X..........] duration: 30.072713397s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:28:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 30.072747873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X..XXX...........XX..X...XX......X..X...X.XXX....X...X..] duration: 5.969271937s trace: https://jaeger.dev.storj.io/trace/795eb7a89d3a1651 FAIL (2025-12-27 13:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ) context deadline exceeded ======LOGS====== Dec 27 13:27:17.774856677 (525h58m17.645939722s) [upload=461359]: Starting upload Dec 27 13:27:17.774949122 (525h58m17.646032198s) [upload=461359]: Done starting upload Dec 27 13:27:17.775049243 (525h58m17.646132308s) [upload=461359]: Uploading segments... Dec 27 13:27:17.775267238 (525h58m17.646350283s) [upload=461359]: Got next segment. Inline: false Dec 27 13:27:17.775307154 (525h58m17.646390199s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:27:17.775313998 (525h58m17.646397033s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:27:18.038975516 (525h58m17.910058561s) [seg_pos={0 0}, upload=461359]: Begin upload segment... Dec 27 13:27:18.039023948 (525h58m17.910106993s) [seg_pos={0 0}, upload=461359]: Done begin upload segment. Dec 27 13:27:18.039056890 (525h58m17.910139935s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039081648 (525h58m17.910164693s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039173783 (525h58m17.910256818s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039612218 (525h58m17.910695263s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039763686 (525h58m17.910846731s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040233381 (525h58m17.911316416s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040247377 (525h58m17.911330422s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040616851 (525h58m17.911699886s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040623193 (525h58m17.911706238s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041115771 (525h58m17.912198816s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041479203 (525h58m17.912562248s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041879445 (525h58m17.91296249s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041941533 (525h58m17.913024568s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042281711 (525h58m17.913364746s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042658880 (525h58m17.913741915s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042758278 (525h58m17.913841313s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:19.749190694 (525h58m19.620273729s) [upload=461359]: Next returned nil segment Dec 27 13:27:19.749207025 (525h58m19.62029006s) [upload=461359]: Waiting for error group managing segments... Dec 27 13:27:20.029249725 (525h58m19.90033276s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.039023607 (525h58m19.910106642s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.043960497 (525h58m19.915043543s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.054911700 (525h58m19.925994735s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.070576198 (525h58m19.941659253s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.073387070 (525h58m19.944470115s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.076402032 (525h58m19.947485067s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.337390148 (525h58m20.208473183s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.340016089 (525h58m20.211099124s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.376636711 (525h58m20.247719746s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.420938058 (525h58m20.292021094s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.468046200 (525h58m20.339129245s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939413045 (525h58m20.81049609s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939437562 (525h58m20.810520607s) [seg_pos={0 0}, upload=461359]: Segment reached optimal threshold of 13 pieces. Dec 27 13:27:20.939695693 (525h58m20.810778748s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939706114 (525h58m20.810789159s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939846972 (525h58m20.810930007s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939857632 (525h58m20.810940677s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939897418 (525h58m20.810980463s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939907246 (525h58m20.810990291s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939931553 (525h58m20.811014598s) [seg_pos={0 0}, upload=461359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:27:20.940016755 (525h58m20.811099801s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:27:20.940023999 (525h58m20.811107044s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:27:21.228670943 (525h58m21.099753978s) [upload=461359]: Done uploading segments... Dec 27 13:27:21.228692945 (525h58m21.09977598s) [upload=461359]: Upload finished. err: <nil> Duration: 5.969304841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.............................X...X...X.........] duration: 38.323636044s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:28:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 38.323666681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X..XXX...........XX..X...XX......X..X...X.XXX....X...X.] duration: 5.951062754s trace: https://jaeger.dev.storj.io/trace/795eb7a89d3a1651 FAIL (2025-12-27 13:27:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ) context deadline exceeded ======LOGS====== Dec 27 13:27:17.774856677 (525h58m17.645939722s) [upload=461359]: Starting upload Dec 27 13:27:17.774949122 (525h58m17.646032198s) [upload=461359]: Done starting upload Dec 27 13:27:17.775049243 (525h58m17.646132308s) [upload=461359]: Uploading segments... Dec 27 13:27:17.775267238 (525h58m17.646350283s) [upload=461359]: Got next segment. Inline: false Dec 27 13:27:17.775307154 (525h58m17.646390199s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:27:17.775313998 (525h58m17.646397033s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:27:18.038975516 (525h58m17.910058561s) [seg_pos={0 0}, upload=461359]: Begin upload segment... Dec 27 13:27:18.039023948 (525h58m17.910106993s) [seg_pos={0 0}, upload=461359]: Done begin upload segment. Dec 27 13:27:18.039056890 (525h58m17.910139935s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039081648 (525h58m17.910164693s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039173783 (525h58m17.910256818s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039612218 (525h58m17.910695263s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039763686 (525h58m17.910846731s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040233381 (525h58m17.911316416s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040247377 (525h58m17.911330422s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040616851 (525h58m17.911699886s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040623193 (525h58m17.911706238s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041115771 (525h58m17.912198816s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041479203 (525h58m17.912562248s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041879445 (525h58m17.91296249s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041941533 (525h58m17.913024568s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042281711 (525h58m17.913364746s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042658880 (525h58m17.913741915s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042758278 (525h58m17.913841313s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:19.749190694 (525h58m19.620273729s) [upload=461359]: Next returned nil segment Dec 27 13:27:19.749207025 (525h58m19.62029006s) [upload=461359]: Waiting for error group managing segments... Dec 27 13:27:20.029249725 (525h58m19.90033276s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.039023607 (525h58m19.910106642s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.043960497 (525h58m19.915043543s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.054911700 (525h58m19.925994735s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.070576198 (525h58m19.941659253s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.073387070 (525h58m19.944470115s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.076402032 (525h58m19.947485067s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.337390148 (525h58m20.208473183s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.340016089 (525h58m20.211099124s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.376636711 (525h58m20.247719746s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.420938058 (525h58m20.292021094s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.468046200 (525h58m20.339129245s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939413045 (525h58m20.81049609s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939437562 (525h58m20.810520607s) [seg_pos={0 0}, upload=461359]: Segment reached optimal threshold of 13 pieces. Dec 27 13:27:20.939695693 (525h58m20.810778748s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939706114 (525h58m20.810789159s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939846972 (525h58m20.810930007s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939857632 (525h58m20.810940677s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939897418 (525h58m20.810980463s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939907246 (525h58m20.810990291s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939931553 (525h58m20.811014598s) [seg_pos={0 0}, upload=461359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:27:20.940016755 (525h58m20.811099801s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:27:20.940023999 (525h58m20.811107044s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:27:21.228670943 (525h58m21.099753978s) [upload=461359]: Done uploading segments... Dec 27 13:27:21.228692945 (525h58m21.09977598s) [upload=461359]: Upload finished. err: <nil> Duration: 5.951076831s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [............X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX....] duration: 3.355005306s trace: https://jaeger.dev.storj.io/trace/1d026087b152a47b FAIL (2025-12-27 13:27:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.355163437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X..X..XXX...........XX..X...XX......X..X...X.XXX....X...X] duration: 30.001884594s trace: https://jaeger.dev.storj.io/trace/795eb7a89d3a1651 FAIL (2025-12-27 13:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ) context deadline exceeded ======LOGS====== Dec 27 13:27:17.774856677 (525h58m17.645939722s) [upload=461359]: Starting upload Dec 27 13:27:17.774949122 (525h58m17.646032198s) [upload=461359]: Done starting upload Dec 27 13:27:17.775049243 (525h58m17.646132308s) [upload=461359]: Uploading segments... Dec 27 13:27:17.775267238 (525h58m17.646350283s) [upload=461359]: Got next segment. Inline: false Dec 27 13:27:17.775307154 (525h58m17.646390199s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:27:17.775313998 (525h58m17.646397033s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:27:18.038975516 (525h58m17.910058561s) [seg_pos={0 0}, upload=461359]: Begin upload segment... Dec 27 13:27:18.039023948 (525h58m17.910106993s) [seg_pos={0 0}, upload=461359]: Done begin upload segment. Dec 27 13:27:18.039056890 (525h58m17.910139935s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039081648 (525h58m17.910164693s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039173783 (525h58m17.910256818s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039612218 (525h58m17.910695263s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.039763686 (525h58m17.910846731s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040233381 (525h58m17.911316416s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040247377 (525h58m17.911330422s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040616851 (525h58m17.911699886s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.040623193 (525h58m17.911706238s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041115771 (525h58m17.912198816s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041479203 (525h58m17.912562248s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041879445 (525h58m17.91296249s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.041941533 (525h58m17.913024568s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042281711 (525h58m17.913364746s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042658880 (525h58m17.913741915s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:18.042758278 (525h58m17.913841313s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Uploading piece... Dec 27 13:27:19.749190694 (525h58m19.620273729s) [upload=461359]: Next returned nil segment Dec 27 13:27:19.749207025 (525h58m19.62029006s) [upload=461359]: Waiting for error group managing segments... Dec 27 13:27:20.029249725 (525h58m19.90033276s) [noise=true , address=145.79.227.16:4910 , piece_id=7W7P37BSDEOUYAXK5STEVUOQF6LRERV62AXHJMB5WNABN2UQ25DA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.039023607 (525h58m19.910106642s) [noise=true , address=145.79.227.49:4339 , piece_id=7HOGAVUPM3TSGZ7BFTJKM42RCU5DSXUJYM5VYBGTBZBUHF27E2YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.043960497 (525h58m19.915043543s) [noise=true , address=145.79.227.11:4730 , piece_id=F3DIDWY6WNRCMDOZTFXDHXRBLBKWBLVWXUZIMTAJHMK7RNE5P23Q, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.054911700 (525h58m19.925994735s) [noise=true , address=145.79.227.41:4085 , piece_id=E5OZHR54HZFPB6FOHFCXXZGIVLMJBXUZKY5DN25ASZMVOLP7VRMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.070576198 (525h58m19.941659253s) [noise=true , address=145.79.227.35:4482 , piece_id=TWYD7MEQTTONXKA6JNLSIQB6ETOK6LZK5HHQCC73TP7WRG3EYXRA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.073387070 (525h58m19.944470115s) [noise=true , address=145.79.227.38:4978 , piece_id=T5SSGWTRSTSKRLQ5QAS2SENAXYQQIETWMLLRTNA3BCF5VC7VZGUA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.076402032 (525h58m19.947485067s) [noise=true , address=145.79.227.45:4231 , piece_id=QDHB4F7ITFNNAIF5UWMZFWFBUSDMVDD5GNTGUDCU7OCQDN4LCOZQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.337390148 (525h58m20.208473183s) [noise=true , address=77.90.31.10:4517 , piece_id=66T3VKWPUXSITFCWPFBSWME24PNA52JRQWIJVTNFFCTDEQITC4LQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.340016089 (525h58m20.211099124s) [noise=true , address=77.90.31.15:4697 , piece_id=AHGOU24ATNIAUJRMTBBK7FJ72Z3ZLKQLRZUZ7BROBICNMTUAW6YA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.376636711 (525h58m20.247719746s) [noise=true , address=77.90.31.19:4844 , piece_id=ADNJ6B3V6XH2MWZPOYCCKWF4KWCYO3PRBLTQBAO7WFVK4LMMK7SQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.420938058 (525h58m20.292021094s) [noise=true , address=45.91.116.11:4120 , piece_id=EXEXLRDKA2COR24GEJC7MKEVHQUGR3KYBCAUPMDR7VWHQRGWJWNQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.468046200 (525h58m20.339129245s) [noise=true , address=45.91.116.14:4189 , piece_id=ZZ3DJWJIOTJF5BYVRE3VFCAIMNKH4VDWIRF5DOKAIWAAQFULX5AA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939413045 (525h58m20.81049609s) [noise=true , address=77.90.31.13:4629 , piece_id=F6INOY7YPN4W4PJPY7BJWROE6B5DSLW6IE5CCJXSGBGJ5IQNMEPA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: <nil> Dec 27 13:27:20.939437562 (525h58m20.810520607s) [seg_pos={0 0}, upload=461359]: Segment reached optimal threshold of 13 pieces. Dec 27 13:27:20.939695693 (525h58m20.810778748s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939706114 (525h58m20.810789159s) [noise=true , address=45.91.116.10:4083 , piece_id=AL6M2MYS5M4VWHUSFVFJW26OZPRKRRM64U44PG33DQFCS4DFAWLQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939846972 (525h58m20.810930007s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939857632 (525h58m20.810940677s) [noise=true , address=45.91.116.8:4046 , piece_id=FBCJ6KWORBC7E5BMIYTMVUTIXQ3ALECPOM53PB3V3KJNG3R2DPMQ, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939897418 (525h58m20.810980463s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 13:27:20.939907246 (525h58m20.810990291s) [noise=true , address=45.91.116.13:4156 , piece_id=GTARF74OXSQHOXIYTEEM6QM6XSETCQUAP2SXTQWIPL3OVK3U3TQA, seg_pos={0 0}, upload=461359]: Long tail cancellation due to optimal threshold Dec 27 13:27:20.939931553 (525h58m20.811014598s) [seg_pos={0 0}, upload=461359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:27:20.940016755 (525h58m20.811099801s) [upload=461359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:27:20.940023999 (525h58m20.811107044s) [upload=461359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:27:21.228670943 (525h58m21.099753978s) [upload=461359]: Done uploading segments... Dec 27 13:27:21.228692945 (525h58m21.09977598s) [upload=461359]: Upload finished. err: <nil> Duration: 30.001907016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.............................X...X...X........] duration: 53.920567359s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 53.920589791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.............X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX...] duration: 4.556545879s trace: https://jaeger.dev.storj.io/trace/1d026087b152a47b FAIL (2025-12-27 13:26:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.556570246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X..X..XXX...........XX..X...XX......X..X...X.XXX....X...] duration: 5.746680615s trace: https://jaeger.dev.storj.io/trace/7a8abd5639b6dedf FAIL (2025-12-27 13:26:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA) context deadline exceeded ======LOGS====== Dec 27 13:24:08.225026980 (525h55m8.096110015s) [upload=461315]: Starting upload Dec 27 13:24:08.225210169 (525h55m8.096293214s) [upload=461315]: Done starting upload Dec 27 13:24:08.225243562 (525h55m8.096326607s) [upload=461315]: Uploading segments... Dec 27 13:24:08.225586185 (525h55m8.09666923s) [upload=461315]: Got next segment. Inline: false Dec 27 13:24:08.225650207 (525h55m8.096733252s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:24:08.225656308 (525h55m8.096739363s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:24:08.471364720 (525h55m8.342447775s) [seg_pos={0 0}, upload=461315]: Begin upload segment... Dec 27 13:24:08.471424373 (525h55m8.342507418s) [seg_pos={0 0}, upload=461315]: Done begin upload segment. Dec 27 13:24:08.471459370 (525h55m8.342542415s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.472501654 (525h55m8.343584689s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473136573 (525h55m8.344219618s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473614524 (525h55m8.344697569s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474257098 (525h55m8.345340143s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474773731 (525h55m8.345856776s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475078130 (525h55m8.346161186s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475762404 (525h55m8.346845439s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.476341477 (525h55m8.347424522s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.477060425 (525h55m8.34814347s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.478745955 (525h55m8.349829s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.480359367 (525h55m8.351442402s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.481892577 (525h55m8.352975622s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.485088092 (525h55m8.356171137s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.489080173 (525h55m8.360163219s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.492500707 (525h55m8.363583742s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:10.200470630 (525h55m10.071553675s) [upload=461315]: Next returned nil segment Dec 27 13:24:10.200485357 (525h55m10.071568392s) [upload=461315]: Waiting for error group managing segments... Dec 27 13:24:10.488943072 (525h55m10.360026117s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489287508 (525h55m10.360370553s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489531152 (525h55m10.360614197s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.498293116 (525h55m10.369376151s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.501672050 (525h55m10.372755105s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.504076369 (525h55m10.375159414s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.505550686 (525h55m10.376633731s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506450890 (525h55m10.377533935s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506663064 (525h55m10.377746109s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.518153494 (525h55m10.389236539s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.520043863 (525h55m10.391126898s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.769674003 (525h55m10.640757038s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844409763 (525h55m10.715492808s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844464436 (525h55m10.715547481s) [seg_pos={0 0}, upload=461315]: Segment reached optimal threshold of 13 pieces. Dec 27 13:24:10.844712189 (525h55m10.715795224s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844724312 (525h55m10.715807377s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.844813391 (525h55m10.715896436s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844841795 (525h55m10.71592483s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845038700 (525h55m10.716121745s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.845048709 (525h55m10.716131774s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845078155 (525h55m10.7161612s) [seg_pos={0 0}, upload=461315]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:24:10.845175741 (525h55m10.716258786s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:24:10.845182764 (525h55m10.716265819s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:24:11.168606159 (525h55m11.039689204s) [upload=461315]: Done uploading segments... Dec 27 13:24:11.168632729 (525h55m11.039715774s) [upload=461315]: Upload finished. err: <nil> Duration: 5.746711193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.............................X...X...X.......] duration: 56.760074757s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:26:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 56.760093393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX..] duration: 5.001162398s trace: https://jaeger.dev.storj.io/trace/1d026087b152a47b FAIL (2025-12-27 13:25:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001321611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X..X..XXX...........XX..X...XX......X..X...X.XXX....X..] duration: 16.554949479s trace: https://jaeger.dev.storj.io/trace/7a8abd5639b6dedf FAIL (2025-12-27 13:25:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA) context deadline exceeded ======LOGS====== Dec 27 13:24:08.225026980 (525h55m8.096110015s) [upload=461315]: Starting upload Dec 27 13:24:08.225210169 (525h55m8.096293214s) [upload=461315]: Done starting upload Dec 27 13:24:08.225243562 (525h55m8.096326607s) [upload=461315]: Uploading segments... Dec 27 13:24:08.225586185 (525h55m8.09666923s) [upload=461315]: Got next segment. Inline: false Dec 27 13:24:08.225650207 (525h55m8.096733252s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:24:08.225656308 (525h55m8.096739363s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:24:08.471364720 (525h55m8.342447775s) [seg_pos={0 0}, upload=461315]: Begin upload segment... Dec 27 13:24:08.471424373 (525h55m8.342507418s) [seg_pos={0 0}, upload=461315]: Done begin upload segment. Dec 27 13:24:08.471459370 (525h55m8.342542415s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.472501654 (525h55m8.343584689s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473136573 (525h55m8.344219618s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473614524 (525h55m8.344697569s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474257098 (525h55m8.345340143s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474773731 (525h55m8.345856776s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475078130 (525h55m8.346161186s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475762404 (525h55m8.346845439s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.476341477 (525h55m8.347424522s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.477060425 (525h55m8.34814347s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.478745955 (525h55m8.349829s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.480359367 (525h55m8.351442402s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.481892577 (525h55m8.352975622s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.485088092 (525h55m8.356171137s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.489080173 (525h55m8.360163219s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.492500707 (525h55m8.363583742s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:10.200470630 (525h55m10.071553675s) [upload=461315]: Next returned nil segment Dec 27 13:24:10.200485357 (525h55m10.071568392s) [upload=461315]: Waiting for error group managing segments... Dec 27 13:24:10.488943072 (525h55m10.360026117s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489287508 (525h55m10.360370553s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489531152 (525h55m10.360614197s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.498293116 (525h55m10.369376151s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.501672050 (525h55m10.372755105s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.504076369 (525h55m10.375159414s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.505550686 (525h55m10.376633731s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506450890 (525h55m10.377533935s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506663064 (525h55m10.377746109s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.518153494 (525h55m10.389236539s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.520043863 (525h55m10.391126898s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.769674003 (525h55m10.640757038s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844409763 (525h55m10.715492808s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844464436 (525h55m10.715547481s) [seg_pos={0 0}, upload=461315]: Segment reached optimal threshold of 13 pieces. Dec 27 13:24:10.844712189 (525h55m10.715795224s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844724312 (525h55m10.715807377s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.844813391 (525h55m10.715896436s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844841795 (525h55m10.71592483s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845038700 (525h55m10.716121745s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.845048709 (525h55m10.716131774s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845078155 (525h55m10.7161612s) [seg_pos={0 0}, upload=461315]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:24:10.845175741 (525h55m10.716258786s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:24:10.845182764 (525h55m10.716265819s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:24:11.168606159 (525h55m11.039689204s) [upload=461315]: Done uploading segments... Dec 27 13:24:11.168632729 (525h55m11.039715774s) [upload=461315]: Upload finished. err: <nil> Duration: 16.554974176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.............................X...X...X......] duration: 52.377927552s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:25:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 52.377952639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX.] duration: 3.592342638s trace: https://jaeger.dev.storj.io/trace/1d026087b152a47b FAIL (2025-12-27 13:25:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592362997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X..X..XXX...........XX..X...XX......X..X...X.XXX....X.] duration: 5.626962692s trace: https://jaeger.dev.storj.io/trace/7a8abd5639b6dedf FAIL (2025-12-27 13:24:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA) context deadline exceeded ======LOGS====== Dec 27 13:24:08.225026980 (525h55m8.096110015s) [upload=461315]: Starting upload Dec 27 13:24:08.225210169 (525h55m8.096293214s) [upload=461315]: Done starting upload Dec 27 13:24:08.225243562 (525h55m8.096326607s) [upload=461315]: Uploading segments... Dec 27 13:24:08.225586185 (525h55m8.09666923s) [upload=461315]: Got next segment. Inline: false Dec 27 13:24:08.225650207 (525h55m8.096733252s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:24:08.225656308 (525h55m8.096739363s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:24:08.471364720 (525h55m8.342447775s) [seg_pos={0 0}, upload=461315]: Begin upload segment... Dec 27 13:24:08.471424373 (525h55m8.342507418s) [seg_pos={0 0}, upload=461315]: Done begin upload segment. Dec 27 13:24:08.471459370 (525h55m8.342542415s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.472501654 (525h55m8.343584689s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473136573 (525h55m8.344219618s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473614524 (525h55m8.344697569s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474257098 (525h55m8.345340143s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474773731 (525h55m8.345856776s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475078130 (525h55m8.346161186s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475762404 (525h55m8.346845439s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.476341477 (525h55m8.347424522s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.477060425 (525h55m8.34814347s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.478745955 (525h55m8.349829s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.480359367 (525h55m8.351442402s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.481892577 (525h55m8.352975622s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.485088092 (525h55m8.356171137s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.489080173 (525h55m8.360163219s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.492500707 (525h55m8.363583742s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:10.200470630 (525h55m10.071553675s) [upload=461315]: Next returned nil segment Dec 27 13:24:10.200485357 (525h55m10.071568392s) [upload=461315]: Waiting for error group managing segments... Dec 27 13:24:10.488943072 (525h55m10.360026117s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489287508 (525h55m10.360370553s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489531152 (525h55m10.360614197s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.498293116 (525h55m10.369376151s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.501672050 (525h55m10.372755105s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.504076369 (525h55m10.375159414s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.505550686 (525h55m10.376633731s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506450890 (525h55m10.377533935s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506663064 (525h55m10.377746109s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.518153494 (525h55m10.389236539s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.520043863 (525h55m10.391126898s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.769674003 (525h55m10.640757038s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844409763 (525h55m10.715492808s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844464436 (525h55m10.715547481s) [seg_pos={0 0}, upload=461315]: Segment reached optimal threshold of 13 pieces. Dec 27 13:24:10.844712189 (525h55m10.715795224s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844724312 (525h55m10.715807377s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.844813391 (525h55m10.715896436s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844841795 (525h55m10.71592483s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845038700 (525h55m10.716121745s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.845048709 (525h55m10.716131774s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845078155 (525h55m10.7161612s) [seg_pos={0 0}, upload=461315]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:24:10.845175741 (525h55m10.716258786s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:24:10.845182764 (525h55m10.716265819s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:24:11.168606159 (525h55m11.039689204s) [upload=461315]: Done uploading segments... Dec 27 13:24:11.168632729 (525h55m11.039715774s) [upload=461315]: Upload finished. err: <nil> Duration: 5.626975126s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX....XX] duration: 5.001405621s trace: https://jaeger.dev.storj.io/trace/1d026087b152a47b FAIL (2025-12-27 13:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001425739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X......X..X..XXX...........XX..X...XX......X..X...X.XXX....X] duration: 30.003574362s trace: https://jaeger.dev.storj.io/trace/7a8abd5639b6dedf FAIL (2025-12-27 13:24:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA) context deadline exceeded ======LOGS====== Dec 27 13:24:08.225026980 (525h55m8.096110015s) [upload=461315]: Starting upload Dec 27 13:24:08.225210169 (525h55m8.096293214s) [upload=461315]: Done starting upload Dec 27 13:24:08.225243562 (525h55m8.096326607s) [upload=461315]: Uploading segments... Dec 27 13:24:08.225586185 (525h55m8.09666923s) [upload=461315]: Got next segment. Inline: false Dec 27 13:24:08.225650207 (525h55m8.096733252s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:24:08.225656308 (525h55m8.096739363s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:24:08.471364720 (525h55m8.342447775s) [seg_pos={0 0}, upload=461315]: Begin upload segment... Dec 27 13:24:08.471424373 (525h55m8.342507418s) [seg_pos={0 0}, upload=461315]: Done begin upload segment. Dec 27 13:24:08.471459370 (525h55m8.342542415s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.472501654 (525h55m8.343584689s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473136573 (525h55m8.344219618s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.473614524 (525h55m8.344697569s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474257098 (525h55m8.345340143s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.474773731 (525h55m8.345856776s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475078130 (525h55m8.346161186s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.475762404 (525h55m8.346845439s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.476341477 (525h55m8.347424522s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.477060425 (525h55m8.34814347s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.478745955 (525h55m8.349829s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.480359367 (525h55m8.351442402s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.481892577 (525h55m8.352975622s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.485088092 (525h55m8.356171137s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.489080173 (525h55m8.360163219s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:08.492500707 (525h55m8.363583742s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Uploading piece... Dec 27 13:24:10.200470630 (525h55m10.071553675s) [upload=461315]: Next returned nil segment Dec 27 13:24:10.200485357 (525h55m10.071568392s) [upload=461315]: Waiting for error group managing segments... Dec 27 13:24:10.488943072 (525h55m10.360026117s) [noise=true , address=145.79.227.42:4122 , piece_id=WFQ3EMPIQQG7E6ZMSTRM6GNVNIKDY4YJ6DRDYIGUCTB5FYNWM6QA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489287508 (525h55m10.360370553s) [noise=true , address=145.79.227.45:4231 , piece_id=GCFUXPHCUYYXN4HBTLLS4KBDYTXRACVPUA2E4UXR7PCC6R7SFKLQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.489531152 (525h55m10.360614197s) [noise=true , address=145.79.227.29:4941 , piece_id=ZLRP2SEYCG6RANUY46TM7QOH4O4NA43MA55VYTZKD2E2QL2FI4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.498293116 (525h55m10.369376151s) [noise=true , address=145.79.227.19:4078 , piece_id=JZ5U4FRCNQ4X7PZ6VQ77QUW5ICCA7T76XRVZHXPXJFKLZN37NJFA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.501672050 (525h55m10.372755105s) [noise=true , address=145.79.227.51:4410 , piece_id=HWEBPG6JMZUKEHHC27HTU6LENYIUQZMEP64UL5M2CKCJOJJLZCWA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.504076369 (525h55m10.375159414s) [noise=true , address=145.79.227.37:4567 , piece_id=CNR6CPOLQETDWYWGZ4UGYXB7ZH7VXMKYJ4U6VRDFSAREIXGRKTQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.505550686 (525h55m10.376633731s) [noise=true , address=145.79.227.50:4374 , piece_id=FJSANLPAIM6SBKZVQHYWZU6M4DJTKWKOBWTKRB3KVQQ56ZM2VKQQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506450890 (525h55m10.377533935s) [noise=true , address=145.79.227.16:4911 , piece_id=OWLOVUGNI6PPBK6JWYZ5ILZNRNUPPE2JMOYKNVKOGE2BMOWSPAGQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.506663064 (525h55m10.377746109s) [noise=true , address=145.79.227.39:4013 , piece_id=TEACDBP6HFUC3BPPBRXHPMSAJ42CNQA3NCUO5NF3IFTJ5WKU4SNA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.518153494 (525h55m10.389236539s) [noise=true , address=145.79.227.11:4732 , piece_id=LYX3LLKEQ2YXCO3MX2CK7NJR7MFD5SD6HL46RXFTIS6ZBP2OS4UQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.520043863 (525h55m10.391126898s) [noise=true , address=145.79.227.48:4301 , piece_id=4DCV7EMOHC6BJAL5TQTAM5SOUOJDKYNR3VFL4XO3KXW652KSCNMQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.769674003 (525h55m10.640757038s) [noise=true , address=45.91.116.10:4081 , piece_id=ZPAIEZU3Z44O6SPC3XX7WZRXSOITGBRLBAQY4YPFKDUW6HV6LZBQ, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844409763 (525h55m10.715492808s) [noise=true , address=45.91.116.11:4120 , piece_id=SA6SFBNDD3GEMTP3UTKXGZRMQVEHERCINIVSSU2FQHT5HR7TY4FA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: <nil> Dec 27 13:24:10.844464436 (525h55m10.715547481s) [seg_pos={0 0}, upload=461315]: Segment reached optimal threshold of 13 pieces. Dec 27 13:24:10.844712189 (525h55m10.715795224s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844724312 (525h55m10.715807377s) [noise=true , address=45.91.116.14:4189 , piece_id=JJJNKLY6CAGNE7BNNHCR7LE7K3WT2MDY7ZFPNZXVUJQZOZDGVYRA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.844813391 (525h55m10.715896436s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.844841795 (525h55m10.71592483s) [noise=true , address=45.91.116.4:4937 , piece_id=KE74UHZWZSDS5KRFAK7ET3IUE6FKEN53BA5NTNNITBS2PMO65JOA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845038700 (525h55m10.716121745s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 13:24:10.845048709 (525h55m10.716131774s) [noise=true , address=45.91.116.13:4155 , piece_id=5OZA2PJ7Y5LMG5EIZJHKBW2JGKBQ7QA54SJ7SRQD3P53A7YSR2EA, seg_pos={0 0}, upload=461315]: Long tail cancellation due to optimal threshold Dec 27 13:24:10.845078155 (525h55m10.7161612s) [seg_pos={0 0}, upload=461315]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:24:10.845175741 (525h55m10.716258786s) [upload=461315]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:24:10.845182764 (525h55m10.716265819s) [upload=461315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:24:11.168606159 (525h55m11.039689204s) [upload=461315]: Done uploading segments... Dec 27 13:24:11.168632729 (525h55m11.039715774s) [upload=461315]: Upload finished. err: <nil> Duration: 30.003601784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.............................X...X...X.....] duration: 50.312365833s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 50.312393346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX....X] duration: 5.001338466s trace: https://jaeger.dev.storj.io/trace/1edfafe70694bd4d FAIL (2025-12-27 13:23:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00136128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X......X..X..XXX...........XX..X...XX......X..X...X.XXX....] duration: 5.616269639s trace: https://jaeger.dev.storj.io/trace/7d436b09cc2948b FAIL (2025-12-27 13:23:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ) context deadline exceeded ======LOGS====== Dec 27 13:19:51.799608640 (525h50m51.670691675s) [upload=461256]: Starting upload Dec 27 13:19:51.799672842 (525h50m51.670755887s) [upload=461256]: Done starting upload Dec 27 13:19:51.799720412 (525h50m51.670803458s) [upload=461256]: Uploading segments... Dec 27 13:19:51.799875729 (525h50m51.670958764s) [upload=461256]: Got next segment. Inline: false Dec 27 13:19:51.799905775 (525h50m51.67098881s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:19:51.799910104 (525h50m51.670993149s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:19:52.045822884 (525h50m51.916905929s) [seg_pos={0 0}, upload=461256]: Begin upload segment... Dec 27 13:19:52.045883801 (525h50m51.916966836s) [seg_pos={0 0}, upload=461256]: Done begin upload segment. Dec 27 13:19:52.045910001 (525h50m51.916993036s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048564515 (525h50m51.91964755s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048943347 (525h50m51.920026382s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049273825 (525h50m51.92035686s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049618612 (525h50m51.920701647s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049964421 (525h50m51.921047466s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050349394 (525h50m51.921432439s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050668110 (525h50m51.921751145s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051171059 (525h50m51.922254104s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051685839 (525h50m51.922768874s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052009866 (525h50m51.923092901s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052333542 (525h50m51.923416577s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052679991 (525h50m51.923763027s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053092848 (525h50m51.924175893s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053450749 (525h50m51.924533794s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053834790 (525h50m51.924917825s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:53.762765173 (525h50m53.633848218s) [upload=461256]: Next returned nil segment Dec 27 13:19:53.762780092 (525h50m53.633863137s) [upload=461256]: Waiting for error group managing segments... Dec 27 13:19:54.047442719 (525h50m53.918525754s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.047551616 (525h50m53.918634661s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.059243630 (525h50m53.930326675s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.082683734 (525h50m53.953766769s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.083521679 (525h50m53.954604714s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.084178300 (525h50m53.955261335s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.259562352 (525h50m54.130645387s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.311220448 (525h50m54.182303483s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.360437085 (525h50m54.23152012s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.398664116 (525h50m54.269747151s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.401916790 (525h50m54.272999835s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.435210947 (525h50m54.306293992s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493267487 (525h50m54.364350522s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493299128 (525h50m54.364382163s) [seg_pos={0 0}, upload=461256]: Segment reached optimal threshold of 13 pieces. Dec 27 13:19:54.493543242 (525h50m54.364626277s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493553642 (525h50m54.364636697s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.493890243 (525h50m54.364973278s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493899280 (525h50m54.364982325s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494348886 (525h50m54.365431961s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: context canceled; context canceled Dec 27 13:19:54.494486769 (525h50m54.365569804s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494520854 (525h50m54.365603889s) [seg_pos={0 0}, upload=461256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:19:54.494666120 (525h50m54.365749165s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:19:54.494673434 (525h50m54.365756469s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:19:54.779819332 (525h50m54.650902367s) [upload=461256]: Done uploading segments... Dec 27 13:19:54.779835663 (525h50m54.650918698s) [upload=461256]: Upload finished. err: <nil> Duration: 5.616302983s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX....] duration: 4.329868035s trace: https://jaeger.dev.storj.io/trace/6bbcb3e2173599e FAIL (2025-12-27 13:23:19 UTC) orbiter warning: context deadline exceeded Duration: 4.329891091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.............................X...X...X....] duration: 46.74868542s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:22:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 46.748706071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X......X..X..XXX...........XX..X...XX......X..X...X.XXX...] duration: 5.92707659s trace: https://jaeger.dev.storj.io/trace/7d436b09cc2948b FAIL (2025-12-27 13:22:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ) context deadline exceeded ======LOGS====== Dec 27 13:19:51.799608640 (525h50m51.670691675s) [upload=461256]: Starting upload Dec 27 13:19:51.799672842 (525h50m51.670755887s) [upload=461256]: Done starting upload Dec 27 13:19:51.799720412 (525h50m51.670803458s) [upload=461256]: Uploading segments... Dec 27 13:19:51.799875729 (525h50m51.670958764s) [upload=461256]: Got next segment. Inline: false Dec 27 13:19:51.799905775 (525h50m51.67098881s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:19:51.799910104 (525h50m51.670993149s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:19:52.045822884 (525h50m51.916905929s) [seg_pos={0 0}, upload=461256]: Begin upload segment... Dec 27 13:19:52.045883801 (525h50m51.916966836s) [seg_pos={0 0}, upload=461256]: Done begin upload segment. Dec 27 13:19:52.045910001 (525h50m51.916993036s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048564515 (525h50m51.91964755s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048943347 (525h50m51.920026382s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049273825 (525h50m51.92035686s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049618612 (525h50m51.920701647s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049964421 (525h50m51.921047466s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050349394 (525h50m51.921432439s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050668110 (525h50m51.921751145s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051171059 (525h50m51.922254104s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051685839 (525h50m51.922768874s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052009866 (525h50m51.923092901s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052333542 (525h50m51.923416577s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052679991 (525h50m51.923763027s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053092848 (525h50m51.924175893s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053450749 (525h50m51.924533794s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053834790 (525h50m51.924917825s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:53.762765173 (525h50m53.633848218s) [upload=461256]: Next returned nil segment Dec 27 13:19:53.762780092 (525h50m53.633863137s) [upload=461256]: Waiting for error group managing segments... Dec 27 13:19:54.047442719 (525h50m53.918525754s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.047551616 (525h50m53.918634661s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.059243630 (525h50m53.930326675s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.082683734 (525h50m53.953766769s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.083521679 (525h50m53.954604714s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.084178300 (525h50m53.955261335s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.259562352 (525h50m54.130645387s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.311220448 (525h50m54.182303483s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.360437085 (525h50m54.23152012s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.398664116 (525h50m54.269747151s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.401916790 (525h50m54.272999835s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.435210947 (525h50m54.306293992s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493267487 (525h50m54.364350522s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493299128 (525h50m54.364382163s) [seg_pos={0 0}, upload=461256]: Segment reached optimal threshold of 13 pieces. Dec 27 13:19:54.493543242 (525h50m54.364626277s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493553642 (525h50m54.364636697s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.493890243 (525h50m54.364973278s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493899280 (525h50m54.364982325s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494348886 (525h50m54.365431961s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: context canceled; context canceled Dec 27 13:19:54.494486769 (525h50m54.365569804s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494520854 (525h50m54.365603889s) [seg_pos={0 0}, upload=461256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:19:54.494666120 (525h50m54.365749165s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:19:54.494673434 (525h50m54.365756469s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:19:54.779819332 (525h50m54.650902367s) [upload=461256]: Done uploading segments... Dec 27 13:19:54.779835663 (525h50m54.650918698s) [upload=461256]: Upload finished. err: <nil> Duration: 5.927105906s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX...] duration: 5.000769679s trace: https://jaeger.dev.storj.io/trace/6bbcb3e2173599e FAIL (2025-12-27 13:22:08 UTC) orbiter warning: context deadline exceeded Duration: 5.000796739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.............................X...X...X...] duration: 46.500351209s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:22:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 46.500378751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X......X..X..XXX...........XX..X...XX......X..X...X.XXX..] duration: 5.85131155s trace: https://jaeger.dev.storj.io/trace/7d436b09cc2948b FAIL (2025-12-27 13:21:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ) context deadline exceeded ======LOGS====== Dec 27 13:19:51.799608640 (525h50m51.670691675s) [upload=461256]: Starting upload Dec 27 13:19:51.799672842 (525h50m51.670755887s) [upload=461256]: Done starting upload Dec 27 13:19:51.799720412 (525h50m51.670803458s) [upload=461256]: Uploading segments... Dec 27 13:19:51.799875729 (525h50m51.670958764s) [upload=461256]: Got next segment. Inline: false Dec 27 13:19:51.799905775 (525h50m51.67098881s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:19:51.799910104 (525h50m51.670993149s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:19:52.045822884 (525h50m51.916905929s) [seg_pos={0 0}, upload=461256]: Begin upload segment... Dec 27 13:19:52.045883801 (525h50m51.916966836s) [seg_pos={0 0}, upload=461256]: Done begin upload segment. Dec 27 13:19:52.045910001 (525h50m51.916993036s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048564515 (525h50m51.91964755s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048943347 (525h50m51.920026382s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049273825 (525h50m51.92035686s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049618612 (525h50m51.920701647s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049964421 (525h50m51.921047466s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050349394 (525h50m51.921432439s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050668110 (525h50m51.921751145s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051171059 (525h50m51.922254104s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051685839 (525h50m51.922768874s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052009866 (525h50m51.923092901s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052333542 (525h50m51.923416577s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052679991 (525h50m51.923763027s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053092848 (525h50m51.924175893s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053450749 (525h50m51.924533794s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053834790 (525h50m51.924917825s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:53.762765173 (525h50m53.633848218s) [upload=461256]: Next returned nil segment Dec 27 13:19:53.762780092 (525h50m53.633863137s) [upload=461256]: Waiting for error group managing segments... Dec 27 13:19:54.047442719 (525h50m53.918525754s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.047551616 (525h50m53.918634661s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.059243630 (525h50m53.930326675s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.082683734 (525h50m53.953766769s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.083521679 (525h50m53.954604714s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.084178300 (525h50m53.955261335s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.259562352 (525h50m54.130645387s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.311220448 (525h50m54.182303483s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.360437085 (525h50m54.23152012s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.398664116 (525h50m54.269747151s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.401916790 (525h50m54.272999835s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.435210947 (525h50m54.306293992s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493267487 (525h50m54.364350522s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493299128 (525h50m54.364382163s) [seg_pos={0 0}, upload=461256]: Segment reached optimal threshold of 13 pieces. Dec 27 13:19:54.493543242 (525h50m54.364626277s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493553642 (525h50m54.364636697s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.493890243 (525h50m54.364973278s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493899280 (525h50m54.364982325s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494348886 (525h50m54.365431961s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: context canceled; context canceled Dec 27 13:19:54.494486769 (525h50m54.365569804s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494520854 (525h50m54.365603889s) [seg_pos={0 0}, upload=461256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:19:54.494666120 (525h50m54.365749165s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:19:54.494673434 (525h50m54.365756469s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:19:54.779819332 (525h50m54.650902367s) [upload=461256]: Done uploading segments... Dec 27 13:19:54.779835663 (525h50m54.650918698s) [upload=461256]: Upload finished. err: <nil> Duration: 5.851344844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.....X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX..] duration: 5.001287718s trace: https://jaeger.dev.storj.io/trace/6bbcb3e2173599e FAIL (2025-12-27 13:21:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001314908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [......X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX.] duration: 5.001071894s trace: https://jaeger.dev.storj.io/trace/6bbcb3e2173599e FAIL (2025-12-27 13:21:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001093524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X......X..X..XXX...........XX..X...XX......X..X...X.XXX.] duration: 5.668554709s trace: https://jaeger.dev.storj.io/trace/7d436b09cc2948b FAIL (2025-12-27 13:21:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ) context deadline exceeded ======LOGS====== Dec 27 13:19:51.799608640 (525h50m51.670691675s) [upload=461256]: Starting upload Dec 27 13:19:51.799672842 (525h50m51.670755887s) [upload=461256]: Done starting upload Dec 27 13:19:51.799720412 (525h50m51.670803458s) [upload=461256]: Uploading segments... Dec 27 13:19:51.799875729 (525h50m51.670958764s) [upload=461256]: Got next segment. Inline: false Dec 27 13:19:51.799905775 (525h50m51.67098881s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:19:51.799910104 (525h50m51.670993149s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:19:52.045822884 (525h50m51.916905929s) [seg_pos={0 0}, upload=461256]: Begin upload segment... Dec 27 13:19:52.045883801 (525h50m51.916966836s) [seg_pos={0 0}, upload=461256]: Done begin upload segment. Dec 27 13:19:52.045910001 (525h50m51.916993036s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048564515 (525h50m51.91964755s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048943347 (525h50m51.920026382s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049273825 (525h50m51.92035686s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049618612 (525h50m51.920701647s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049964421 (525h50m51.921047466s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050349394 (525h50m51.921432439s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050668110 (525h50m51.921751145s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051171059 (525h50m51.922254104s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051685839 (525h50m51.922768874s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052009866 (525h50m51.923092901s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052333542 (525h50m51.923416577s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052679991 (525h50m51.923763027s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053092848 (525h50m51.924175893s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053450749 (525h50m51.924533794s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053834790 (525h50m51.924917825s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:53.762765173 (525h50m53.633848218s) [upload=461256]: Next returned nil segment Dec 27 13:19:53.762780092 (525h50m53.633863137s) [upload=461256]: Waiting for error group managing segments... Dec 27 13:19:54.047442719 (525h50m53.918525754s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.047551616 (525h50m53.918634661s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.059243630 (525h50m53.930326675s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.082683734 (525h50m53.953766769s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.083521679 (525h50m53.954604714s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.084178300 (525h50m53.955261335s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.259562352 (525h50m54.130645387s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.311220448 (525h50m54.182303483s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.360437085 (525h50m54.23152012s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.398664116 (525h50m54.269747151s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.401916790 (525h50m54.272999835s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.435210947 (525h50m54.306293992s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493267487 (525h50m54.364350522s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493299128 (525h50m54.364382163s) [seg_pos={0 0}, upload=461256]: Segment reached optimal threshold of 13 pieces. Dec 27 13:19:54.493543242 (525h50m54.364626277s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493553642 (525h50m54.364636697s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.493890243 (525h50m54.364973278s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493899280 (525h50m54.364982325s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494348886 (525h50m54.365431961s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: context canceled; context canceled Dec 27 13:19:54.494486769 (525h50m54.365569804s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494520854 (525h50m54.365603889s) [seg_pos={0 0}, upload=461256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:19:54.494666120 (525h50m54.365749165s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:19:54.494673434 (525h50m54.365756469s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:19:54.779819332 (525h50m54.650902367s) [upload=461256]: Done uploading segments... Dec 27 13:19:54.779835663 (525h50m54.650918698s) [upload=461256]: Upload finished. err: <nil> Duration: 5.668595416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.............................X...X...X..] duration: 44.393351953s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:20:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 44.393370128s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X......X..............X..X.X...X.XX..XX.X.......XXXXX..XXXXX] duration: 5.000678986s trace: https://jaeger.dev.storj.io/trace/6bbcb3e2173599e FAIL (2025-12-27 13:20:26 UTC) orbiter warning: context deadline exceeded Duration: 5.000708112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX.X......X..X..XXX...........XX..X...XX......X..X...X.XXX] duration: 30.003874103s trace: https://jaeger.dev.storj.io/trace/7d436b09cc2948b FAIL (2025-12-27 13:20:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ) context deadline exceeded ======LOGS====== Dec 27 13:19:51.799608640 (525h50m51.670691675s) [upload=461256]: Starting upload Dec 27 13:19:51.799672842 (525h50m51.670755887s) [upload=461256]: Done starting upload Dec 27 13:19:51.799720412 (525h50m51.670803458s) [upload=461256]: Uploading segments... Dec 27 13:19:51.799875729 (525h50m51.670958764s) [upload=461256]: Got next segment. Inline: false Dec 27 13:19:51.799905775 (525h50m51.67098881s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:19:51.799910104 (525h50m51.670993149s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:19:52.045822884 (525h50m51.916905929s) [seg_pos={0 0}, upload=461256]: Begin upload segment... Dec 27 13:19:52.045883801 (525h50m51.916966836s) [seg_pos={0 0}, upload=461256]: Done begin upload segment. Dec 27 13:19:52.045910001 (525h50m51.916993036s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048564515 (525h50m51.91964755s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.048943347 (525h50m51.920026382s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049273825 (525h50m51.92035686s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049618612 (525h50m51.920701647s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.049964421 (525h50m51.921047466s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050349394 (525h50m51.921432439s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.050668110 (525h50m51.921751145s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051171059 (525h50m51.922254104s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.051685839 (525h50m51.922768874s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052009866 (525h50m51.923092901s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052333542 (525h50m51.923416577s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.052679991 (525h50m51.923763027s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053092848 (525h50m51.924175893s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053450749 (525h50m51.924533794s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:52.053834790 (525h50m51.924917825s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Uploading piece... Dec 27 13:19:53.762765173 (525h50m53.633848218s) [upload=461256]: Next returned nil segment Dec 27 13:19:53.762780092 (525h50m53.633863137s) [upload=461256]: Waiting for error group managing segments... Dec 27 13:19:54.047442719 (525h50m53.918525754s) [noise=true , address=145.79.227.52:4447 , piece_id=NB7BKQPPXIYXPQR5EYYUHQHLD7353RVJT7OCTXQWT6EBN4FFMJHA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.047551616 (525h50m53.918634661s) [noise=true , address=145.79.227.11:4732 , piece_id=PY5MHAWMJ3JVTAST6SKJLJJ55EVJIFQMZ5X7GDPVW6D67WL43MNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.059243630 (525h50m53.930326675s) [noise=true , address=145.79.227.36:4517 , piece_id=2KCYYRZEVK7KS54J6QF4ONZSHU7N4YYDVIVHKXDCJMJBTN3GY3JA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.082683734 (525h50m53.953766769s) [noise=true , address=145.79.227.51:4410 , piece_id=2NEDPVCF22GZPJ76ME7E4DJRR47FVRGNV33VFD2TZJUVENDHNQOQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.083521679 (525h50m53.954604714s) [noise=true , address=145.79.227.5:4221 , piece_id=JUCRABWHRVDPEIIONOKAYWD3RTUVXBQ5IARKQY56LQA6S35G5UOA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.084178300 (525h50m53.955261335s) [noise=true , address=145.79.227.39:4014 , piece_id=UPIL2YA7AGC7QGBEOZLCFFT5JLWPWJNC5OP5DQQDVBY2CLC5BCNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.259562352 (525h50m54.130645387s) [noise=true , address=77.90.31.13:4629 , piece_id=ZOHJVDMDUL3JG3RNY3B6OBSTRJBRSZ3AIV5JGJYHC7XCZ3RPAOCQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.311220448 (525h50m54.182303483s) [noise=true , address=45.91.116.14:4192 , piece_id=VZOB2D2WWYRNMR4B5FJSJR5MADAYTNDBM62PUFALB7JGLXUSBD5A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.360437085 (525h50m54.23152012s) [noise=true , address=77.90.31.18:4808 , piece_id=ODSD4TJ6HJKWXVL525FJLZOM3ZCOWLMK5D3CFKAYPNX2HRFQW5FQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.398664116 (525h50m54.269747151s) [noise=true , address=45.91.116.7:4010 , piece_id=K63JOIKMEFQ2WICSZRVYM5RWK6MBMUFHUS2RUYSPMIKWUXMGIQ2A, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.401916790 (525h50m54.272999835s) [noise=true , address=45.91.116.5:4297 , piece_id=MUTZ2VPP7JZ77GPIJCST2QXSJUIBYJCVXTUYBCLLHHORNJVN3WVQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.435210947 (525h50m54.306293992s) [noise=true , address=77.90.31.10:4520 , piece_id=WE6IDO5IW5FPNBP5BXVVVJYHHS6URZZOFQEPCZFYPCK6MHJK2Q5Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493267487 (525h50m54.364350522s) [noise=true , address=77.90.31.11:4553 , piece_id=VDQOPVGSOFBYR2NE6TDXJAM22MTJJBX4IYARE2BDBCVUGPD6CJNA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: <nil> Dec 27 13:19:54.493299128 (525h50m54.364382163s) [seg_pos={0 0}, upload=461256]: Segment reached optimal threshold of 13 pieces. Dec 27 13:19:54.493543242 (525h50m54.364626277s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493553642 (525h50m54.364636697s) [noise=true , address=77.90.31.12:4590 , piece_id=PTOZROM5LG57OYI3W2VBXI2HEYWHJ3TRFNZ6BDKI5UV43HBILSGQ, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.493890243 (525h50m54.364973278s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:54.493899280 (525h50m54.364982325s) [noise=true , address=45.91.116.8:4046 , piece_id=35AHEBBNBQQXXND6TVRG6BTE7RYA5SYGWEMKFW63MBKWSYMZNT4Q, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494348886 (525h50m54.365431961s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: context canceled; context canceled Dec 27 13:19:54.494486769 (525h50m54.365569804s) [noise=true , address=45.91.116.13:4153 , piece_id=N3XCF2MFKBI6GCPBUVNOGNXF5PUAAYA55J6VHZPJ7AZNNM7TDWBA, seg_pos={0 0}, upload=461256]: Long tail cancellation due to optimal threshold Dec 27 13:19:54.494520854 (525h50m54.365603889s) [seg_pos={0 0}, upload=461256]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:19:54.494666120 (525h50m54.365749165s) [upload=461256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:19:54.494673434 (525h50m54.365756469s) [upload=461256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:19:54.779819332 (525h50m54.650902367s) [upload=461256]: Done uploading segments... Dec 27 13:19:54.779835663 (525h50m54.650918698s) [upload=461256]: Upload finished. err: <nil> Duration: 30.003885775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.............................X...X...X.] duration: 56.533086742s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:20:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 56.533101811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X......X..............X..X.X...X.XX..XX.X.......XXXXX..XXXX] duration: 5.001890815s trace: https://jaeger.dev.storj.io/trace/1073c643bbb039a1 FAIL (2025-12-27 13:20:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001922615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..XX.X......X..X..XXX...........XX..X...XX......X..X...X.XX] duration: 30.002536655s trace: https://jaeger.dev.storj.io/trace/4b6828c6fe530e31 FAIL (2025-12-27 13:19:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: APM6DV2LABTVRPV4UPACYN3MVPLHUEOZUCNDT2FUQ3U77W5CJYZQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 3KORUNTKKYGQQLKQ3EWYIKSJZST2K4JUNJVNOU3XFYWMSEL7EQQQ) context deadline exceeded ======LOGS====== 27 13:19:21.579674992 (525h50m21.450758037s) [upload=461249]: Starting upload Dec 27 13:19:21.579746649 (525h50m21.450829684s) [upload=461249]: Done starting upload Dec 27 13:19:21.579788358 (525h50m21.450871393s) [upload=461249]: Uploading segments... Dec 27 13:19:21.579992076 (525h50m21.451075111s) [upload=461249]: Got next segment. Inline: false Dec 27 13:19:21.580024217 (525h50m21.451107262s) [upload=461249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:19:21.580028966 (525h50m21.451112001s) [upload=461249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:19:21.865029337 (525h50m21.736112382s) [seg_pos={0 0}, upload=461249]: Begin upload segment... Dec 27 13:19:21.865096345 (525h50m21.73617939s) [seg_pos={0 0}, upload=461249]: Done begin upload segment. Dec 27 13:19:21.865127855 (525h50m21.73621089s) [noise=true , address=145.79.227.29:4941 , piece_id=4MFMR35KG335BFSKKCKUW6NYVFKGJ5ZREKSO25ZDPE5CHVTV6QIQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.865186245 (525h50m21.73626929s) [noise=true , address=145.79.227.51:4411 , piece_id=57JKCQ6UPHFKC7OAGGXWA5EJT5JDSTHBTD27JEMCMDN2YLSOFJ3A, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.865859738 (525h50m21.736942783s) [noise=true , address=45.91.116.10:4084 , piece_id=BFACMK7QPJ4CH7G7XL66O6Q5EVVQHRGIZMQKRIWEDN6RNADRNSKQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.865921727 (525h50m21.737004762s) [noise=true , address=145.79.227.8:4329 , piece_id=ENPCGIQVCZLIWNGLYA55GZPREKACTBX5MT6XIKYKAPMDRAJ26WHA, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.866290168 (525h50m21.737373203s) [noise=true , address=45.91.116.6:4976 , piece_id=Q7XYHZPEALEEM7ZLY6FV3W5DEEKJACGY6A2CCYVRNANCLNLZAQOA, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.866429523 (525h50m21.737512568s) [noise=true , address=45.91.116.13:4156 , piece_id=KVPZB5S5ADKPPWRAAANQD26PXZUG56TIDXAXA32M2HBLSVELFHWA, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.866587524 (525h50m21.737670559s) [noise=true , address=145.79.227.7:4294 , piece_id=5QRRZ6XDECAXB6UPBVBDHHYG5PU3MPJHJF3JQ3NLNOW7GKPMD2XA, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.866894008 (525h50m21.737977053s) [noise=true , address=145.79.227.48:4302 , piece_id=XLCSAQ7RLR22WP5C2GKYZTWLUDZFBK4Z6LKZLWEQRZVXEEGZA75Q, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.866916630 (525h50m21.737999665s) [noise=true , address=145.79.227.27:4690 , piece_id=L7CEON6XRS3FMO3VUJTPD7XAG6VFIVY3QYQUZB45GOL7TCKFTDOQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.867260415 (525h50m21.73834346s) [noise=true , address=45.91.116.9:4264 , piece_id=APM6DV2LABTVRPV4UPACYN3MVPLHUEOZUCNDT2FUQ3U77W5CJYZQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.867361267 (525h50m21.738444312s) [noise=true , address=145.79.227.52:4445 , piece_id=PLH2ETU2FLUYUW5D5OIKIIF7TKGNOZTT5WU5NZL3IERTSIJN6GZQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.867836432 (525h50m21.738919468s) [noise=true , address=145.79.227.20:4438 , piece_id=ZGTRAMGH4MCRSPE35WIRPPNZGFSIDUVSCKJE4T2XROSZPNV7QAZA, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.867869896 (525h50m21.738952941s) [noise=true , address=145.79.227.39:4014 , piece_id=3KVCRRC4SROMD22IHNO7IWS45ASZWD6VKXY3JREHIRRSNSYZL4SQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.868224892 (525h50m21.739307927s) [noise=true , address=77.90.31.10:4519 , piece_id=3KORUNTKKYGQQLKQ3EWYIKSJZST2K4JUNJVNOU3XFYWMSEL7EQQQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.868575079 (525h50m21.739658124s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYAVKOP3P3EPQCRKLBH526VODGYF5LKCBBGJBH5MPP3SGIAT4YAQ, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:21.869509077 (525h50m21.740592122s) [noise=true , address=145.79.227.10:4403 , piece_id=NE76BVDMRE47IMJSBYYZUJE72HSD4UJFJDLLK5I3BNHAI2G66KZA, seg_pos={0 0}, upload=461249]: Uploading piece... Dec 27 13:19:23.472915453 (525h50m23.343998488s) [upload=461249]: Next returned nil segment Dec 27 13:19:23.472934129 (525h50m23.344017164s) [upload=461249]: Waiting for error group managing segments... Dec 27 13:19:23.859437199 (525h50m23.730520244s) [noise=true , address=145.79.227.8:4329 , piece_id=ENPCGIQVCZLIWNGLYA55GZPREKACTBX5MT6XIKYKAPMDRAJ26WHA, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.863171439 (525h50m23.734254474s) [noise=true , address=145.79.227.48:4302 , piece_id=XLCSAQ7RLR22WP5C2GKYZTWLUDZFBK4Z6LKZLWEQRZVXEEGZA75Q, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.871891563 (525h50m23.742974608s) [noise=true , address=145.79.227.52:4445 , piece_id=PLH2ETU2FLUYUW5D5OIKIIF7TKGNOZTT5WU5NZL3IERTSIJN6GZQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.871972819 (525h50m23.743055854s) [noise=true , address=145.79.227.39:4014 , piece_id=3KVCRRC4SROMD22IHNO7IWS45ASZWD6VKXY3JREHIRRSNSYZL4SQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.881353411 (525h50m23.752436456s) [noise=true , address=145.79.227.10:4403 , piece_id=NE76BVDMRE47IMJSBYYZUJE72HSD4UJFJDLLK5I3BNHAI2G66KZA, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.886860036 (525h50m23.757943071s) [noise=true , address=145.79.227.7:4294 , piece_id=5QRRZ6XDECAXB6UPBVBDHHYG5PU3MPJHJF3JQ3NLNOW7GKPMD2XA, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.889074162 (525h50m23.760157217s) [noise=true , address=145.79.227.29:4941 , piece_id=4MFMR35KG335BFSKKCKUW6NYVFKGJ5ZREKSO25ZDPE5CHVTV6QIQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.895968029 (525h50m23.767051065s) [noise=true , address=145.79.227.20:4438 , piece_id=ZGTRAMGH4MCRSPE35WIRPPNZGFSIDUVSCKJE4T2XROSZPNV7QAZA, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.900306261 (525h50m23.771389296s) [noise=true , address=145.79.227.27:4690 , piece_id=L7CEON6XRS3FMO3VUJTPD7XAG6VFIVY3QYQUZB45GOL7TCKFTDOQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:23.901168742 (525h50m23.772251787s) [noise=true , address=145.79.227.51:4411 , piece_id=57JKCQ6UPHFKC7OAGGXWA5EJT5JDSTHBTD27JEMCMDN2YLSOFJ3A, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:24.149008562 (525h50m24.020091607s) [noise=true , address=45.91.116.6:4976 , piece_id=Q7XYHZPEALEEM7ZLY6FV3W5DEEKJACGY6A2CCYVRNANCLNLZAQOA, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:24.256481390 (525h50m24.127564425s) [noise=true , address=45.91.116.9:4264 , piece_id=APM6DV2LABTVRPV4UPACYN3MVPLHUEOZUCNDT2FUQ3U77W5CJYZQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:24.387099370 (525h50m24.258182415s) [noise=true , address=77.90.31.10:4519 , piece_id=3KORUNTKKYGQQLKQ3EWYIKSJZST2K4JUNJVNOU3XFYWMSEL7EQQQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: <nil> Dec 27 13:19:24.387118946 (525h50m24.258201981s) [seg_pos={0 0}, upload=461249]: Segment reached optimal threshold of 13 pieces. Dec 27 13:19:24.387304990 (525h50m24.258388025s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYAVKOP3P3EPQCRKLBH526VODGYF5LKCBBGJBH5MPP3SGIAT4YAQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:35680->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:35680->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->77.90.31.21:4878: use of closed network connection Dec 27 13:19:24.387310512 (525h50m24.258393557s) [noise=true , address=77.90.31.21:4878 , piece_id=ZYAVKOP3P3EPQCRKLBH526VODGYF5LKCBBGJBH5MPP3SGIAT4YAQ, seg_pos={0 0}, upload=461249]: Long tail cancellation due to optimal threshold Dec 27 13:19:24.387505352 (525h50m24.258588397s) [noise=true , address=45.91.116.13:4156 , piece_id=KVPZB5S5ADKPPWRAAANQD26PXZUG56TIDXAXA32M2HBLSVELFHWA, seg_pos={0 0}, upload=461249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:24.387528947 (525h50m24.258612012s) [noise=true , address=45.91.116.13:4156 , piece_id=KVPZB5S5ADKPPWRAAANQD26PXZUG56TIDXAXA32M2HBLSVELFHWA, seg_pos={0 0}, upload=461249]: Long tail cancellation due to optimal threshold Dec 27 13:19:24.387554987 (525h50m24.258638022s) [noise=true , address=45.91.116.10:4084 , piece_id=BFACMK7QPJ4CH7G7XL66O6Q5EVVQHRGIZMQKRIWEDN6RNADRNSKQ, seg_pos={0 0}, upload=461249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 13:19:24.387563543 (525h50m24.258646578s) [noise=true , address=45.91.116.10:4084 , piece_id=BFACMK7QPJ4CH7G7XL66O6Q5EVVQHRGIZMQKRIWEDN6RNADRNSKQ, seg_pos={0 0}, upload=461249]: Long tail cancellation due to optimal threshold Dec 27 13:19:24.387673352 (525h50m24.258756387s) [seg_pos={0 0}, upload=461249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:19:24.387758233 (525h50m24.258841278s) [upload=461249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:19:24.387763414 (525h50m24.258846449s) [upload=461249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:19:24.691968494 (525h50m24.563051539s) [upload=461249]: Done uploading segments... Dec 27 13:19:24.691989133 (525h50m24.563072168s) [upload=461249]: Upload finished. err: <nil> Duration: 30.002559969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.............................X...X...X] duration: 1m0.002999401s trace: https://jaeger.dev.storj.io/trace/30bf78cd8c90f5f0 FAIL (2025-12-27 13:19:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, Piece ID: EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q) context deadline exceeded; piecestore: (Node ID: 12FNxEAEnP3DqdDb16XaBCdmoqdUPEmMtLHDoTuQDBukr2nQWDW, Piece ID: 6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ) context deadline exceeded; piecestore: (Node ID: 12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, Piece ID: EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA) context deadline exceeded ======LOGS====== se=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.188780617 (525h49m14.059863662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189075448 (525h49m14.060158493s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189489046 (525h49m14.060572091s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.189884169 (525h49m14.060967214s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190343603 (525h49m14.061426648s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.190673611 (525h49m14.061756646s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.192537841 (525h49m14.063620886s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.193023266 (525h49m14.064106311s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.194541586 (525h49m14.065624631s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.196239630 (525h49m14.067322695s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.198646423 (525h49m14.069729468s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.200198899 (525h49m14.071281934s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.201434221 (525h49m14.072517276s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.202334736 (525h49m14.073417781s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:14.203607209 (525h49m14.074690254s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Uploading piece... Dec 27 13:18:15.514224089 (525h49m15.385307134s) [upload=461229]: Next returned nil segment Dec 27 13:18:15.514240241 (525h49m15.385323276s) [upload=461229]: Waiting for error group managing segments... Dec 27 13:18:15.849950490 (525h49m15.721033525s) [noise=true , address=64.49.13.170:4724 , piece_id=7DJ5EGAZNT5GXD3FF3IM7ILIGQ3V7TKMANVO3A4MEN6HUUF66RNQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.067157801 (525h49m15.938240846s) [noise=true , address=145.79.227.10:4429 , piece_id=OGHESAQRKBPQFF7L6PGPZ3OWKWXXVWJV2PY5Y3XMQHALWWTFV3XA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.218088177 (525h49m16.089171222s) [noise=true , address=193.149.164.143:4695 , piece_id=EKTMVOBCCMKCETM3QCBX3LZDDL5RZBZVTNKK2JWXOGSGZGVZFHKA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.222802204 (525h49m16.093885239s) [noise=true , address=77.90.31.17:4797 , piece_id=6XGLB2LASCRJKM5A5NRX5QOYENX7QGLF7ZM6U33ZCBYBLJWYWOEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.233614783 (525h49m16.104697828s) [noise=true , address=77.90.31.19:4872 , piece_id=SUQUATJCFTBJJ3RU5S4YO3TZJPBACO5TRPE3WV5DBN7XENOAH6QQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.263577988 (525h49m16.134661033s) [noise=true , address=145.79.227.34:4914 , piece_id=ABDV6DEBROX6GSNSCP6JQTU3QK6AKJ3QRQNXP5DEDE3Q3ZWVANFQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.319486407 (525h49m16.190569442s) [noise=true , address=77.90.31.21:4883 , piece_id=JPKCJRPX2PBXCV4EUON2OVFFIAR3MUAMU5TMN4JEWJRXQPKBI7TA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:16.437484266 (525h49m16.308567301s) [noise=true , address=77.90.31.13:4648 , piece_id=DMD4A2Q4LLZJFJIRN52HWLFHTUXNZBLQA5XHZQIY7AJIICOP4APQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.025977688 (525h49m16.897060724s) [noise=true , address=145.79.227.44:4200 , piece_id=GSHS5AR6CRUQY6BMPNSKKYQYEWZIDTCJVR6GJYGFRJ33UBI27LOA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.081484353 (525h49m16.952567398s) [noise=true , address=45.91.116.10:4108 , piece_id=BW4JAL34OK5NKNRK2X645RN4QN5V2ZBCAJWZKM2ZYAZIDFZHJWMA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.292684976 (525h49m17.163768031s) [noise=true , address=45.91.116.8:4061 , piece_id=W3WOE5QZD3M2UPTBZQETNRGWNRGPAVA6TND5ID2XXXO4GHLDOMHA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.364756081 (525h49m17.235839126s) [noise=true , address=145.79.227.40:4076 , piece_id=IP3ZVKFFSWJSBWQXA4HNGAXRMDM7OAOCBUYR2M4TZBIFHCJ6OZEQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.390569834 (525h49m17.261652869s) [noise=true , address=45.91.116.9:4277 , piece_id=VZVGW6AXW6XXYR5RST6HK52NJI5ERN46NUSERQVSEOGNTDMCAD7A, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498717376 (525h49m17.369800411s) [noise=true , address=77.90.31.11:4587 , piece_id=EEFUGLTEBZPZSHXETJYR2VMIXD3YKPHUF54XG65THDNZTGIQE66Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: <nil> Dec 27 13:18:17.498735371 (525h49m17.369818416s) [seg_pos={0 0}, upload=461229]: Segment reached optimal threshold of 14 pieces. Dec 27 13:18:17.498952925 (525h49m17.37003597s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.498960138 (525h49m17.370043174s) [noise=true , address=77.90.31.18:4827 , piece_id=3GMVKQZ2FOQS4OV4UM5WWSH7LMJJB635Q762YHMAHW7GFN7LEDVA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499134230 (525h49m17.370217285s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499153617 (525h49m17.370236662s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58850->64.49.13.169:4707: use of closed network connection Dec 27 13:18:17.499162243 (525h49m17.370245289s) [noise=true , address=64.49.13.169:4707 , piece_id=D76AJSQ4DANEJ7PFE6ITCIPMS2JZBM5KA7VFHH3V3OPCWNTXDZ4Q, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499170329 (525h49m17.370253374s) [noise=true , address=77.90.31.15:4701 , piece_id=PAKLOR65AMECM332ZM564HSPPPQL5LTWOLA6J5NUQ7YSWWPPAJQA, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499368927 (525h49m17.370451972s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:17.499388886 (525h49m17.370471921s) [noise=true , address=77.90.31.12:4617 , piece_id=RE2SSABMRCSPQ3T4ORW4M74JKO6C32VDVYR67B57OML3TIU3LVRQ, seg_pos={0 0}, upload=461229]: Long tail cancellation due to optimal threshold Dec 27 13:18:17.499526658 (525h49m17.370609693s) [seg_pos={0 0}, upload=461229]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:18:17.499602362 (525h49m17.370685407s) [upload=461229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:17.499607221 (525h49m17.370690266s) [upload=461229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:17.795410571 (525h49m17.666493606s) [upload=461229]: Done uploading segments... Dec 27 13:18:17.795431832 (525h49m17.666514867s) [upload=461229]: Upload finished. err: <nil> Duration: 1m0.003011142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.X......X..............X..X.X...X.XX..XX.X.......XXXXX..XXX] duration: 5.001533665s trace: https://jaeger.dev.storj.io/trace/1fcf67d594d47df0 FAIL (2025-12-27 13:19:12 UTC) orbiter warning: context deadline exceeded Duration: 5.00155818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..XX.X......X..X..XXX...........XX..X...XX......X..X...X.X] duration: 30.002514509s trace: https://jaeger.dev.storj.io/trace/47bfa67babe7e68e FAIL (2025-12-27 13:19:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 7CI7R5TE24PPDYK3XMUPJZKWUFBQHKKHQGXYMZYZH5OLAACQ74IQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: YPRHXMH5XUOLBSHEWVUPX5XZHGHT7JPK5XJ5H3DWDL4K4A63YTDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: WLVM6NAMTBURZZXQPO7HLQQYQRJFGERN6NTID7UXX6TJIP2LKSOA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: IJRPAYMANBV5LHE46K2LX2XUWDUMIJJ4INAN3IHANT3ZAJDBQ5CQ) context deadline exceeded ======LOGS====== (525h49m39.776167915s) [upload=461240]: Done starting upload Dec 27 13:18:39.905181544 (525h49m39.776264579s) [upload=461240]: Uploading segments... Dec 27 13:18:39.905280573 (525h49m39.776363608s) [upload=461240]: Got next segment. Inline: false Dec 27 13:18:39.905327401 (525h49m39.776410436s) [upload=461240]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:18:39.905332391 (525h49m39.776415436s) [upload=461240]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:18:40.183294639 (525h49m40.054377684s) [seg_pos={0 0}, upload=461240]: Begin upload segment... Dec 27 13:18:40.183409839 (525h49m40.054492884s) [seg_pos={0 0}, upload=461240]: Done begin upload segment. Dec 27 13:18:40.183524808 (525h49m40.054607843s) [noise=true , address=145.79.227.17:4971 , piece_id=MDUXY4FZLCCRK5UXDIOPFSVJ2KHDKRG2GKX22BP33QREXT2KFKRQ, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.183537712 (525h49m40.054620757s) [noise=true , address=145.79.227.13:4802 , piece_id=TRQZTOUVLRHYCIBUBZ7WMWGEZLDZVFQBSHXCBZY7KHMPB2E5MLQA, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.184518330 (525h49m40.055601375s) [noise=true , address=77.90.31.21:4880 , piece_id=IJRPAYMANBV5LHE46K2LX2XUWDUMIJJ4INAN3IHANT3ZAJDBQ5CQ, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.185019514 (525h49m40.056102549s) [noise=true , address=145.79.227.10:4402 , piece_id=LH2BBCTIDC67IQMALQDVDBS772CSJ7BJSI4BM66XNC34D2XFMI7Q, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.185148490 (525h49m40.056231535s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZLGYY4DZ4T3QWTGGUH7FA5G6SFUD63I2345IJLWAJ64YN7ZDNDQ, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.187174628 (525h49m40.058257663s) [noise=true , address=145.79.227.47:4265 , piece_id=NNDADU5M2PT34SAGNKVZ4LYK7CH2QIY2VVXEYM5XYKPV2BZHZYMA, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.187902243 (525h49m40.058985278s) [noise=true , address=45.91.116.11:4118 , piece_id=SEVA6YCJIVKWQFXJTSVSU7DLJ7CFU3BQFSBVWQYMXUCUSHC2RZ7Q, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.188686075 (525h49m40.05976912s) [noise=true , address=45.91.116.4:4940 , piece_id=4FR6CSCPKWDQ7BVG2KERJTTFOVT4WHWMTT3S3OQN5GYNFZ7YBA4Q, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.189639231 (525h49m40.060722266s) [noise=true , address=145.79.227.35:4482 , piece_id=X73P3RQL4GMTZK2VKTVYOCOFVIHZ7G3J3UJH55MO4OC5GZPIABXA, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.190519817 (525h49m40.061602852s) [noise=true , address=145.79.227.46:4603 , piece_id=2DPGCV5ZDZGIT324AN5YKQK5FUGZW2CKSASOCYNTFCZJECG6EPPA, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.191247453 (525h49m40.062330488s) [noise=true , address=77.90.31.11:4555 , piece_id=FRTDSUCJFRIJ33JPFRJXUL7TGN4FW423SIB37VC6OR6RKCU777NQ, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.193239386 (525h49m40.064322431s) [noise=true , address=45.91.116.10:4083 , piece_id=7CI7R5TE24PPDYK3XMUPJZKWUFBQHKKHQGXYMZYZH5OLAACQ74IQ, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.201475088 (525h49m40.072558123s) [noise=true , address=77.90.31.16:4735 , piece_id=YPRHXMH5XUOLBSHEWVUPX5XZHGHT7JPK5XJ5H3DWDL4K4A63YTDQ, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.203684295 (525h49m40.07476733s) [noise=true , address=45.91.116.7:4012 , piece_id=WLVM6NAMTBURZZXQPO7HLQQYQRJFGERN6NTID7UXX6TJIP2LKSOA, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.205436602 (525h49m40.076519647s) [noise=true , address=145.79.227.48:4302 , piece_id=QXTGZP6JK32CUOTMXQCVATUT4R3IWHNTDTJLO3PKLBGHVM3RIQ5Q, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:40.209138711 (525h49m40.080221756s) [noise=true , address=145.79.227.3:4149 , piece_id=NKRZOI6Q5VT6DRPQAW6M4YI3CZGKX7EKH3V2CLBPWJ6LZPZIZOIA, seg_pos={0 0}, upload=461240]: Uploading piece... Dec 27 13:18:41.900860541 (525h49m41.771943586s) [upload=461240]: Next returned nil segment Dec 27 13:18:41.900889516 (525h49m41.771972561s) [upload=461240]: Waiting for error group managing segments... Dec 27 13:18:42.183866423 (525h49m42.054949458s) [noise=true , address=145.79.227.17:4971 , piece_id=MDUXY4FZLCCRK5UXDIOPFSVJ2KHDKRG2GKX22BP33QREXT2KFKRQ, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.185697981 (525h49m42.056781026s) [noise=true , address=145.79.227.10:4402 , piece_id=LH2BBCTIDC67IQMALQDVDBS772CSJ7BJSI4BM66XNC34D2XFMI7Q, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.196806103 (525h49m42.067889138s) [noise=true , address=145.79.227.46:4603 , piece_id=2DPGCV5ZDZGIT324AN5YKQK5FUGZW2CKSASOCYNTFCZJECG6EPPA, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.204984225 (525h49m42.07606726s) [noise=true , address=145.79.227.35:4482 , piece_id=X73P3RQL4GMTZK2VKTVYOCOFVIHZ7G3J3UJH55MO4OC5GZPIABXA, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.215690561 (525h49m42.086773596s) [noise=true , address=145.79.227.47:4265 , piece_id=NNDADU5M2PT34SAGNKVZ4LYK7CH2QIY2VVXEYM5XYKPV2BZHZYMA, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.216803400 (525h49m42.087886446s) [noise=true , address=145.79.227.13:4802 , piece_id=TRQZTOUVLRHYCIBUBZ7WMWGEZLDZVFQBSHXCBZY7KHMPB2E5MLQA, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.222327029 (525h49m42.093410064s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZLGYY4DZ4T3QWTGGUH7FA5G6SFUD63I2345IJLWAJ64YN7ZDNDQ, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.233650450 (525h49m42.104733485s) [noise=true , address=145.79.227.48:4302 , piece_id=QXTGZP6JK32CUOTMXQCVATUT4R3IWHNTDTJLO3PKLBGHVM3RIQ5Q, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.243219161 (525h49m42.114302196s) [noise=true , address=145.79.227.3:4149 , piece_id=NKRZOI6Q5VT6DRPQAW6M4YI3CZGKX7EKH3V2CLBPWJ6LZPZIZOIA, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.525023825 (525h49m42.39610686s) [noise=true , address=77.90.31.16:4735 , piece_id=YPRHXMH5XUOLBSHEWVUPX5XZHGHT7JPK5XJ5H3DWDL4K4A63YTDQ, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:42.576259627 (525h49m42.447342672s) [noise=true , address=45.91.116.7:4012 , piece_id=WLVM6NAMTBURZZXQPO7HLQQYQRJFGERN6NTID7UXX6TJIP2LKSOA, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:43.150204918 (525h49m43.021287954s) [noise=true , address=45.91.116.10:4083 , piece_id=7CI7R5TE24PPDYK3XMUPJZKWUFBQHKKHQGXYMZYZH5OLAACQ74IQ, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:43.195300879 (525h49m43.066383924s) [noise=true , address=77.90.31.21:4880 , piece_id=IJRPAYMANBV5LHE46K2LX2XUWDUMIJJ4INAN3IHANT3ZAJDBQ5CQ, seg_pos={0 0}, upload=461240]: Done uploading piece. err: <nil> Dec 27 13:18:43.195320736 (525h49m43.066403771s) [seg_pos={0 0}, upload=461240]: Segment reached optimal threshold of 13 pieces. Dec 27 13:18:43.195488145 (525h49m43.06657119s) [noise=true , address=45.91.116.4:4940 , piece_id=4FR6CSCPKWDQ7BVG2KERJTTFOVT4WHWMTT3S3OQN5GYNFZ7YBA4Q, seg_pos={0 0}, upload=461240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:51884->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:51884->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51884->45.91.116.4:4940: use of closed network connection Dec 27 13:18:43.195493785 (525h49m43.06657683s) [noise=true , address=45.91.116.4:4940 , piece_id=4FR6CSCPKWDQ7BVG2KERJTTFOVT4WHWMTT3S3OQN5GYNFZ7YBA4Q, seg_pos={0 0}, upload=461240]: Long tail cancellation due to optimal threshold Dec 27 13:18:43.195782997 (525h49m43.066866032s) [noise=true , address=45.91.116.11:4118 , piece_id=SEVA6YCJIVKWQFXJTSVSU7DLJ7CFU3BQFSBVWQYMXUCUSHC2RZ7Q, seg_pos={0 0}, upload=461240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:58396->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 13:18:43.195789088 (525h49m43.066872123s) [noise=true , address=45.91.116.11:4118 , piece_id=SEVA6YCJIVKWQFXJTSVSU7DLJ7CFU3BQFSBVWQYMXUCUSHC2RZ7Q, seg_pos={0 0}, upload=461240]: Long tail cancellation due to optimal threshold Dec 27 13:18:43.195889389 (525h49m43.066972434s) [noise=true , address=77.90.31.11:4555 , piece_id=FRTDSUCJFRIJ33JPFRJXUL7TGN4FW423SIB37VC6OR6RKCU777NQ, seg_pos={0 0}, upload=461240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 13:18:43.195894579 (525h49m43.066977624s) [noise=true , address=77.90.31.11:4555 , piece_id=FRTDSUCJFRIJ33JPFRJXUL7TGN4FW423SIB37VC6OR6RKCU777NQ, seg_pos={0 0}, upload=461240]: Long tail cancellation due to optimal threshold Dec 27 13:18:43.195911651 (525h49m43.066994686s) [seg_pos={0 0}, upload=461240]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:18:43.195973609 (525h49m43.067056654s) [upload=461240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:18:43.195978649 (525h49m43.067061694s) [upload=461240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:18:43.498608599 (525h49m43.369691634s) [upload=461240]: Done uploading segments... Dec 27 13:18:43.498628567 (525h49m43.369711602s) [upload=461240]: Upload finished. err: <nil> Duration: 30.002538745s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X.X......X..............X..X.X...X.XX..XX.X.......XXXXX..XX] duration: 5.00067144s trace: https://jaeger.dev.storj.io/trace/5cd54e946dc720e5 FAIL (2025-12-27 13:18:54 UTC) orbiter warning: context deadline exceeded Duration: 5.000691899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.............................X...X...] duration: 54.893084221s trace: https://jaeger.dev.storj.io/trace/3fff9d1180db5d69 FAIL (2025-12-27 13:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: 4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ) context deadline exceeded; piecestore: (Node ID: 1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu, Piece ID: 2NBPJXVK52L2M7HNYEH3V5XMA5TJ3O6C27PON23W5QLXYU3S6PAA) context deadline exceeded ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=WYPVV5ND6727W3RUHKVEBW7TY2SLK44OHGIWKNQS3Z7VRA7R6BHA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.313890357 (525h45m29.184973402s) [noise=true , address=77.90.31.19:4874 , piece_id=SXRHPQ6ZKSYS7XMDFM4NM444R3UD77XDCZCLDJGYNVXMC5ZSYWAQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314231116 (525h45m29.185314151s) [noise=true , address=45.91.116.5:4329 , piece_id=4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314397303 (525h45m29.185480338s) [noise=true , address=77.90.31.15:4712 , piece_id=PGSHIFGJWOYKM6E7PWUDTUZP6S32PETNZLD73GL3ZMIZP4REYQMA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314590300 (525h45m29.185673345s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314728854 (525h45m29.185811889s) [noise=true , address=145.79.227.40:4062 , piece_id=PN5ZDTWCSQEQXCRG7Q2KWJ35JA26CIVQCEXIPYH7MSEFCW55JAXA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314946217 (525h45m29.186029262s) [noise=true , address=145.79.227.48:4311 , piece_id=JPY7E33FC6AWCECY6IQEXPMRUSCR6EDJXRFARMU7ZWXS5D7RRVUQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315053011 (525h45m29.186136046s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315309129 (525h45m29.186392174s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315374393 (525h45m29.186457428s) [noise=true , address=45.91.116.4:4964 , piece_id=PS273B6BPO4O6J6G4LQLCVRAOKP67PKW4NZRR5FPN5RPCB2MEQQQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315805594 (525h45m29.186888629s) [noise=true , address=145.79.227.44:4225 , piece_id=45QAKVCQ5FPOLE5DRCG2WSFIOYO7C7NH4HTYHSQILOUFM7EFXO2Q, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316139150 (525h45m29.187222185s) [noise=true , address=77.90.31.21:4894 , piece_id=GR5PRAIVWGPPSNDO2IKB7VN65UBASLHZELBLHORDGYOQA4LWZRSA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316474948 (525h45m29.187557983s) [noise=true , address=145.79.227.30:4768 , piece_id=DQ23KUHNR6GG5RA44UOSKTCAIAQWISDVT547E55LOM5RMHRO6PHQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316597522 (525h45m29.187680627s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316807552 (525h45m29.187890587s) [noise=true , address=145.79.227.17:4998 , piece_id=7GPON22DFOGVWIFAFSKV5JCWRUGVYAWGSKPHOXU7ACH6AG3TEUJA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.317160294 (525h45m29.188243329s) [noise=true , address=77.90.31.12:4613 , piece_id=UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:30.687966688 (525h45m30.559049733s) [upload=461177]: Next returned nil segment Dec 27 13:14:30.687980184 (525h45m30.559063219s) [upload=461177]: Waiting for error group managing segments... Dec 27 13:14:31.040556219 (525h45m30.911639264s) [noise=true , address=193.149.164.143:4701 , piece_id=WYPVV5ND6727W3RUHKVEBW7TY2SLK44OHGIWKNQS3Z7VRA7R6BHA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.181412965 (525h45m31.05249601s) [noise=true , address=145.79.227.48:4311 , piece_id=JPY7E33FC6AWCECY6IQEXPMRUSCR6EDJXRFARMU7ZWXS5D7RRVUQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.185708925 (525h45m31.056791961s) [noise=true , address=145.79.227.40:4062 , piece_id=PN5ZDTWCSQEQXCRG7Q2KWJ35JA26CIVQCEXIPYH7MSEFCW55JAXA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.220843017 (525h45m31.091926062s) [noise=true , address=145.79.227.17:4998 , piece_id=7GPON22DFOGVWIFAFSKV5JCWRUGVYAWGSKPHOXU7ACH6AG3TEUJA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.303221950 (525h45m31.174304995s) [noise=true , address=145.79.227.52:4478 , piece_id=TPSQDF6VYSA7UY4GOJBXN5HX7NGMHKXOS35ICXIUVQCDZYILISBA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.396094786 (525h45m31.267177831s) [noise=true , address=77.90.31.12:4613 , piece_id=UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.467160147 (525h45m31.338243192s) [noise=true , address=77.90.31.14:4687 , piece_id=2NBPJXVK52L2M7HNYEH3V5XMA5TJ3O6C27PON23W5QLXYU3S6PAA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.494978408 (525h45m31.366061453s) [noise=true , address=77.90.31.19:4874 , piece_id=SXRHPQ6ZKSYS7XMDFM4NM444R3UD77XDCZCLDJGYNVXMC5ZSYWAQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.550392295 (525h45m31.42147534s) [noise=true , address=45.91.116.4:4964 , piece_id=PS273B6BPO4O6J6G4LQLCVRAOKP67PKW4NZRR5FPN5RPCB2MEQQQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.731063575 (525h45m31.60214661s) [noise=true , address=45.91.116.5:4329 , piece_id=4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.811147498 (525h45m31.682230543s) [noise=true , address=77.90.31.15:4712 , piece_id=PGSHIFGJWOYKM6E7PWUDTUZP6S32PETNZLD73GL3ZMIZP4REYQMA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.070197935 (525h45m31.94128098s) [noise=true , address=145.79.227.44:4225 , piece_id=45QAKVCQ5FPOLE5DRCG2WSFIOYO7C7NH4HTYHSQILOUFM7EFXO2Q, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.081701990 (525h45m31.952785025s) [noise=true , address=145.79.227.30:4768 , piece_id=DQ23KUHNR6GG5RA44UOSKTCAIAQWISDVT547E55LOM5RMHRO6PHQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.087629396 (525h45m31.958712432s) [noise=true , address=77.90.31.21:4894 , piece_id=GR5PRAIVWGPPSNDO2IKB7VN65UBASLHZELBLHORDGYOQA4LWZRSA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.087648273 (525h45m31.958731318s) [seg_pos={0 0}, upload=461177]: Segment reached optimal threshold of 14 pieces. Dec 27 13:14:32.087908097 (525h45m31.958991142s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection Dec 27 13:14:32.087914629 (525h45m31.958997664s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.087964986 (525h45m31.959048021s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.087980906 (525h45m31.959063941s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088048234 (525h45m31.959131279s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.088053634 (525h45m31.959136669s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088203389 (525h45m31.959286434s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.088208870 (525h45m31.959291905s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088227615 (525h45m31.95931065s) [seg_pos={0 0}, upload=461177]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:14:32.088294693 (525h45m31.959377728s) [upload=461177]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:14:32.088299031 (525h45m31.959382066s) [upload=461177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:14:32.389880444 (525h45m32.260963489s) [upload=461177]: Done uploading segments... Dec 27 13:14:32.389899390 (525h45m32.260982426s) [upload=461177]: Upload finished. err: <nil> Duration: 54.893097666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..X.X......X..............X..X.X...X.XX..XX.X.......XXXXX..X] duration: 5.001543791s trace: https://jaeger.dev.storj.io/trace/485572df7fd359fc FAIL (2025-12-27 13:18:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001570681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...X.X......X..............X..X.X...X.XX..XX.X.......XXXXX..] duration: 5.001221177s trace: https://jaeger.dev.storj.io/trace/3aebcbc9f18d5e0f FAIL (2025-12-27 13:17:39 UTC) orbiter warning: context deadline exceeded Duration: 5.00124874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X..XX.X......X..X..XXX...........XX..X...XX......X..X...X.] duration: 5.7596145s trace: https://jaeger.dev.storj.io/trace/6dcc678d5e444950 FAIL (2025-12-27 13:17:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: PN2BXPBD5BWORRWLG52YQLR5OYQ6O4WGORV7O3RZMTVV5KTRBNCA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: T27KKADTN4WUWERSMZNSJ2DSQDVLWXJKGOZJBAVY27CKAJRHFU4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: BJBBPRAUSSSNQKGCBLCXYIXUMHMO5BRN35W7H6RLM2DBIJLVZPNA) context deadline exceeded ======LOGS====== pload Dec 27 13:16:58.186032541 (525h47m58.057115596s) [upload=461212]: Done starting upload Dec 27 13:16:58.186182086 (525h47m58.057265121s) [upload=461212]: Uploading segments... Dec 27 13:16:58.186549335 (525h47m58.05763237s) [upload=461212]: Got next segment. Inline: false Dec 27 13:16:58.186628656 (525h47m58.057711701s) [upload=461212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:16:58.186635068 (525h47m58.057718113s) [upload=461212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:16:58.437476294 (525h47m58.308559329s) [seg_pos={0 0}, upload=461212]: Begin upload segment... Dec 27 13:16:58.437526499 (525h47m58.308609544s) [seg_pos={0 0}, upload=461212]: Done begin upload segment. Dec 27 13:16:58.437569140 (525h47m58.308652175s) [noise=true , address=145.79.227.19:4078 , piece_id=AGHXD2YPJ5ORZWI3CPU7IE44HNPPZBSUNIKB7ZRBPHMHYOQQSMTA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.437576043 (525h47m58.308659078s) [noise=true , address=45.91.116.8:4045 , piece_id=BJBBPRAUSSSNQKGCBLCXYIXUMHMO5BRN35W7H6RLM2DBIJLVZPNA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.437729947 (525h47m58.308812992s) [noise=true , address=145.79.227.23:4546 , piece_id=IEF7PC3GTW5T2WO5VID2VKXK63AGQUW3FN6J6EQ5R4BVS34TX5KQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.438696798 (525h47m58.309779833s) [noise=true , address=45.91.116.4:4937 , piece_id=ONXH4NBXFS6LESNLDQ2ST3YVLA5B7EGNZ2WPZVQ4R2CZZU52N4CQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439008582 (525h47m58.310091617s) [noise=true , address=77.90.31.16:4736 , piece_id=ZDSZRDAYSNNRVCQMCEBJPOSO36BEW2S4IFZJ2DZI2B4DC3RWX42A, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439290228 (525h47m58.310373263s) [noise=true , address=45.91.116.6:4973 , piece_id=AMRZE6G7XDVZIRQ42IEP3IUTVWLX4JID4YVY3UBJFLTYHJY6GYMQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439625446 (525h47m58.310708481s) [noise=true , address=145.79.227.38:4977 , piece_id=VWZMOQHOUKKS7F4VUYZNNV4ZNE3WKK4ZRLP7VINAEYZ7RI47OOLQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439919136 (525h47m58.311002171s) [noise=true , address=77.90.31.17:4770 , piece_id=6KW52BFIVBAJ22EGESQ3KBYNEOF6VCRJJMHOIRKDM5CXM46BKKIA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.440345567 (525h47m58.311428602s) [noise=true , address=77.90.31.14:4661 , piece_id=DEVJK4PB5NQTGN5ZQ5EMN2DKA5AWJPFTVVGMJRW3YCGECPYYKNPQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.441424452 (525h47m58.312507497s) [noise=true , address=45.91.116.14:4189 , piece_id=5C7PJ343S76P6HNYJQXW2YYSJUGK5BY76PNFOSAKXGH64D33OF5A, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.445288349 (525h47m58.316371394s) [noise=true , address=77.90.31.13:4629 , piece_id=O5YIVNCFDOTVCMV2TXHXVSF72EIURGNUH63F6SIG7LRD7FUCFE7Q, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.447433585 (525h47m58.31851663s) [noise=true , address=45.91.116.10:4083 , piece_id=PN2BXPBD5BWORRWLG52YQLR5OYQ6O4WGORV7O3RZMTVV5KTRBNCA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.454578069 (525h47m58.325661134s) [noise=true , address=145.79.227.45:4231 , piece_id=JRBZROBWTERURCHMRSXEVSPO7UGQJ6W3JTUR4YTNLQVEU7VALYHA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.456034973 (525h47m58.327118008s) [noise=true , address=145.79.227.4:4185 , piece_id=XKSYVHEL2EHRNIM6KZFV2CNUVM7RLRZ7GQN7QYVOSFQ27XYRA2SQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.457353253 (525h47m58.328436288s) [noise=true , address=45.91.116.11:4120 , piece_id=T27KKADTN4WUWERSMZNSJ2DSQDVLWXJKGOZJBAVY27CKAJRHFU4Q, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.461655094 (525h47m58.332738129s) [noise=true , address=45.91.116.5:4300 , piece_id=Z63XTB4CSF365SNHSLPTWPSWWJ7NJ5YH2EG65HZVBANUAU3LLK7Q, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:17:00.165338771 (525h48m0.036421806s) [upload=461212]: Next returned nil segment Dec 27 13:17:00.165352547 (525h48m0.036435592s) [upload=461212]: Waiting for error group managing segments... Dec 27 13:17:00.450830316 (525h48m0.321913351s) [noise=true , address=145.79.227.38:4977 , piece_id=VWZMOQHOUKKS7F4VUYZNNV4ZNE3WKK4ZRLP7VINAEYZ7RI47OOLQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.454955922 (525h48m0.326038967s) [noise=true , address=145.79.227.23:4546 , piece_id=IEF7PC3GTW5T2WO5VID2VKXK63AGQUW3FN6J6EQ5R4BVS34TX5KQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.467928403 (525h48m0.339011448s) [noise=true , address=145.79.227.4:4185 , piece_id=XKSYVHEL2EHRNIM6KZFV2CNUVM7RLRZ7GQN7QYVOSFQ27XYRA2SQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.481620256 (525h48m0.352703291s) [noise=true , address=145.79.227.19:4078 , piece_id=AGHXD2YPJ5ORZWI3CPU7IE44HNPPZBSUNIKB7ZRBPHMHYOQQSMTA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.488524973 (525h48m0.359608008s) [noise=true , address=145.79.227.45:4231 , piece_id=JRBZROBWTERURCHMRSXEVSPO7UGQJ6W3JTUR4YTNLQVEU7VALYHA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.738011138 (525h48m0.609094183s) [noise=true , address=77.90.31.16:4736 , piece_id=ZDSZRDAYSNNRVCQMCEBJPOSO36BEW2S4IFZJ2DZI2B4DC3RWX42A, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.744063212 (525h48m0.615146247s) [noise=true , address=45.91.116.11:4120 , piece_id=T27KKADTN4WUWERSMZNSJ2DSQDVLWXJKGOZJBAVY27CKAJRHFU4Q, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.773135911 (525h48m0.644218946s) [noise=true , address=45.91.116.8:4045 , piece_id=BJBBPRAUSSSNQKGCBLCXYIXUMHMO5BRN35W7H6RLM2DBIJLVZPNA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.919321624 (525h48m0.790404669s) [noise=true , address=45.91.116.14:4189 , piece_id=5C7PJ343S76P6HNYJQXW2YYSJUGK5BY76PNFOSAKXGH64D33OF5A, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.945998622 (525h48m0.817081667s) [noise=true , address=77.90.31.14:4661 , piece_id=DEVJK4PB5NQTGN5ZQ5EMN2DKA5AWJPFTVVGMJRW3YCGECPYYKNPQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.543141494 (525h48m1.414224539s) [noise=true , address=45.91.116.4:4937 , piece_id=ONXH4NBXFS6LESNLDQ2ST3YVLA5B7EGNZ2WPZVQ4R2CZZU52N4CQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.608630186 (525h48m1.479713231s) [noise=true , address=77.90.31.13:4629 , piece_id=O5YIVNCFDOTVCMV2TXHXVSF72EIURGNUH63F6SIG7LRD7FUCFE7Q, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.632968480 (525h48m1.504051535s) [noise=true , address=45.91.116.10:4083 , piece_id=PN2BXPBD5BWORRWLG52YQLR5OYQ6O4WGORV7O3RZMTVV5KTRBNCA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.633000511 (525h48m1.504083556s) [seg_pos={0 0}, upload=461212]: Segment reached optimal threshold of 13 pieces. Dec 27 13:17:01.633530531 (525h48m1.504613586s) [noise=true , address=45.91.116.5:4300 , piece_id=Z63XTB4CSF365SNHSLPTWPSWWJ7NJ5YH2EG65HZVBANUAU3LLK7Q, seg_pos={0 0}, upload=461212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:33096->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:33096->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33096->45.91.116.5:4300: use of closed network connection Dec 27 13:17:01.633542113 (525h48m1.504625168s) [noise=true , address=45.91.116.5:4300 , piece_id=Z63XTB4CSF365SNHSLPTWPSWWJ7NJ5YH2EG65HZVBANUAU3LLK7Q, seg_pos={0 0}, upload=461212]: Long tail cancellation due to optimal threshold Dec 27 13:17:01.634250391 (525h48m1.505333446s) [noise=true , address=45.91.116.6:4973 , piece_id=AMRZE6G7XDVZIRQ42IEP3IUTVWLX4JID4YVY3UBJFLTYHJY6GYMQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:34800->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 13:17:01.634261443 (525h48m1.505344488s) [noise=true , address=45.91.116.6:4973 , piece_id=AMRZE6G7XDVZIRQ42IEP3IUTVWLX4JID4YVY3UBJFLTYHJY6GYMQ, seg_pos={0 0}, upload=461212]: Long tail cancellation due to optimal threshold Dec 27 13:17:01.634703674 (525h48m1.505786719s) [noise=true , address=77.90.31.17:4770 , piece_id=6KW52BFIVBAJ22EGESQ3KBYNEOF6VCRJJMHOIRKDM5CXM46BKKIA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Dec 27 13:17:01.634715157 (525h48m1.505798202s) [noise=true , address=77.90.31.17:4770 , piece_id=6KW52BFIVBAJ22EGESQ3KBYNEOF6VCRJJMHOIRKDM5CXM46BKKIA, seg_pos={0 0}, upload=461212]: Long tail cancellation due to optimal threshold Dec 27 13:17:01.634741967 (525h48m1.505825012s) [seg_pos={0 0}, upload=461212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:17:01.634870803 (525h48m1.505953848s) [upload=461212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:17:01.634879749 (525h48m1.505962794s) [upload=461212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:17:01.940383913 (525h48m1.811466968s) [upload=461212]: Done uploading segments... Dec 27 13:17:01.940413299 (525h48m1.811496354s) [upload=461212]: Upload finished. err: <nil> Duration: 5.759643985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X..XX.X......X..X..XXX...........XX..X...XX......X..X...X] duration: 30.003408446s trace: https://jaeger.dev.storj.io/trace/6dcc678d5e444950 FAIL (2025-12-27 13:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: PN2BXPBD5BWORRWLG52YQLR5OYQ6O4WGORV7O3RZMTVV5KTRBNCA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: T27KKADTN4WUWERSMZNSJ2DSQDVLWXJKGOZJBAVY27CKAJRHFU4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: BJBBPRAUSSSNQKGCBLCXYIXUMHMO5BRN35W7H6RLM2DBIJLVZPNA) context deadline exceeded ======LOGS====== pload Dec 27 13:16:58.186032541 (525h47m58.057115596s) [upload=461212]: Done starting upload Dec 27 13:16:58.186182086 (525h47m58.057265121s) [upload=461212]: Uploading segments... Dec 27 13:16:58.186549335 (525h47m58.05763237s) [upload=461212]: Got next segment. Inline: false Dec 27 13:16:58.186628656 (525h47m58.057711701s) [upload=461212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:16:58.186635068 (525h47m58.057718113s) [upload=461212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:16:58.437476294 (525h47m58.308559329s) [seg_pos={0 0}, upload=461212]: Begin upload segment... Dec 27 13:16:58.437526499 (525h47m58.308609544s) [seg_pos={0 0}, upload=461212]: Done begin upload segment. Dec 27 13:16:58.437569140 (525h47m58.308652175s) [noise=true , address=145.79.227.19:4078 , piece_id=AGHXD2YPJ5ORZWI3CPU7IE44HNPPZBSUNIKB7ZRBPHMHYOQQSMTA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.437576043 (525h47m58.308659078s) [noise=true , address=45.91.116.8:4045 , piece_id=BJBBPRAUSSSNQKGCBLCXYIXUMHMO5BRN35W7H6RLM2DBIJLVZPNA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.437729947 (525h47m58.308812992s) [noise=true , address=145.79.227.23:4546 , piece_id=IEF7PC3GTW5T2WO5VID2VKXK63AGQUW3FN6J6EQ5R4BVS34TX5KQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.438696798 (525h47m58.309779833s) [noise=true , address=45.91.116.4:4937 , piece_id=ONXH4NBXFS6LESNLDQ2ST3YVLA5B7EGNZ2WPZVQ4R2CZZU52N4CQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439008582 (525h47m58.310091617s) [noise=true , address=77.90.31.16:4736 , piece_id=ZDSZRDAYSNNRVCQMCEBJPOSO36BEW2S4IFZJ2DZI2B4DC3RWX42A, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439290228 (525h47m58.310373263s) [noise=true , address=45.91.116.6:4973 , piece_id=AMRZE6G7XDVZIRQ42IEP3IUTVWLX4JID4YVY3UBJFLTYHJY6GYMQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439625446 (525h47m58.310708481s) [noise=true , address=145.79.227.38:4977 , piece_id=VWZMOQHOUKKS7F4VUYZNNV4ZNE3WKK4ZRLP7VINAEYZ7RI47OOLQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.439919136 (525h47m58.311002171s) [noise=true , address=77.90.31.17:4770 , piece_id=6KW52BFIVBAJ22EGESQ3KBYNEOF6VCRJJMHOIRKDM5CXM46BKKIA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.440345567 (525h47m58.311428602s) [noise=true , address=77.90.31.14:4661 , piece_id=DEVJK4PB5NQTGN5ZQ5EMN2DKA5AWJPFTVVGMJRW3YCGECPYYKNPQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.441424452 (525h47m58.312507497s) [noise=true , address=45.91.116.14:4189 , piece_id=5C7PJ343S76P6HNYJQXW2YYSJUGK5BY76PNFOSAKXGH64D33OF5A, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.445288349 (525h47m58.316371394s) [noise=true , address=77.90.31.13:4629 , piece_id=O5YIVNCFDOTVCMV2TXHXVSF72EIURGNUH63F6SIG7LRD7FUCFE7Q, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.447433585 (525h47m58.31851663s) [noise=true , address=45.91.116.10:4083 , piece_id=PN2BXPBD5BWORRWLG52YQLR5OYQ6O4WGORV7O3RZMTVV5KTRBNCA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.454578069 (525h47m58.325661134s) [noise=true , address=145.79.227.45:4231 , piece_id=JRBZROBWTERURCHMRSXEVSPO7UGQJ6W3JTUR4YTNLQVEU7VALYHA, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.456034973 (525h47m58.327118008s) [noise=true , address=145.79.227.4:4185 , piece_id=XKSYVHEL2EHRNIM6KZFV2CNUVM7RLRZ7GQN7QYVOSFQ27XYRA2SQ, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.457353253 (525h47m58.328436288s) [noise=true , address=45.91.116.11:4120 , piece_id=T27KKADTN4WUWERSMZNSJ2DSQDVLWXJKGOZJBAVY27CKAJRHFU4Q, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:16:58.461655094 (525h47m58.332738129s) [noise=true , address=45.91.116.5:4300 , piece_id=Z63XTB4CSF365SNHSLPTWPSWWJ7NJ5YH2EG65HZVBANUAU3LLK7Q, seg_pos={0 0}, upload=461212]: Uploading piece... Dec 27 13:17:00.165338771 (525h48m0.036421806s) [upload=461212]: Next returned nil segment Dec 27 13:17:00.165352547 (525h48m0.036435592s) [upload=461212]: Waiting for error group managing segments... Dec 27 13:17:00.450830316 (525h48m0.321913351s) [noise=true , address=145.79.227.38:4977 , piece_id=VWZMOQHOUKKS7F4VUYZNNV4ZNE3WKK4ZRLP7VINAEYZ7RI47OOLQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.454955922 (525h48m0.326038967s) [noise=true , address=145.79.227.23:4546 , piece_id=IEF7PC3GTW5T2WO5VID2VKXK63AGQUW3FN6J6EQ5R4BVS34TX5KQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.467928403 (525h48m0.339011448s) [noise=true , address=145.79.227.4:4185 , piece_id=XKSYVHEL2EHRNIM6KZFV2CNUVM7RLRZ7GQN7QYVOSFQ27XYRA2SQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.481620256 (525h48m0.352703291s) [noise=true , address=145.79.227.19:4078 , piece_id=AGHXD2YPJ5ORZWI3CPU7IE44HNPPZBSUNIKB7ZRBPHMHYOQQSMTA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.488524973 (525h48m0.359608008s) [noise=true , address=145.79.227.45:4231 , piece_id=JRBZROBWTERURCHMRSXEVSPO7UGQJ6W3JTUR4YTNLQVEU7VALYHA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.738011138 (525h48m0.609094183s) [noise=true , address=77.90.31.16:4736 , piece_id=ZDSZRDAYSNNRVCQMCEBJPOSO36BEW2S4IFZJ2DZI2B4DC3RWX42A, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.744063212 (525h48m0.615146247s) [noise=true , address=45.91.116.11:4120 , piece_id=T27KKADTN4WUWERSMZNSJ2DSQDVLWXJKGOZJBAVY27CKAJRHFU4Q, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.773135911 (525h48m0.644218946s) [noise=true , address=45.91.116.8:4045 , piece_id=BJBBPRAUSSSNQKGCBLCXYIXUMHMO5BRN35W7H6RLM2DBIJLVZPNA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.919321624 (525h48m0.790404669s) [noise=true , address=45.91.116.14:4189 , piece_id=5C7PJ343S76P6HNYJQXW2YYSJUGK5BY76PNFOSAKXGH64D33OF5A, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:00.945998622 (525h48m0.817081667s) [noise=true , address=77.90.31.14:4661 , piece_id=DEVJK4PB5NQTGN5ZQ5EMN2DKA5AWJPFTVVGMJRW3YCGECPYYKNPQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.543141494 (525h48m1.414224539s) [noise=true , address=45.91.116.4:4937 , piece_id=ONXH4NBXFS6LESNLDQ2ST3YVLA5B7EGNZ2WPZVQ4R2CZZU52N4CQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.608630186 (525h48m1.479713231s) [noise=true , address=77.90.31.13:4629 , piece_id=O5YIVNCFDOTVCMV2TXHXVSF72EIURGNUH63F6SIG7LRD7FUCFE7Q, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.632968480 (525h48m1.504051535s) [noise=true , address=45.91.116.10:4083 , piece_id=PN2BXPBD5BWORRWLG52YQLR5OYQ6O4WGORV7O3RZMTVV5KTRBNCA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: <nil> Dec 27 13:17:01.633000511 (525h48m1.504083556s) [seg_pos={0 0}, upload=461212]: Segment reached optimal threshold of 13 pieces. Dec 27 13:17:01.633530531 (525h48m1.504613586s) [noise=true , address=45.91.116.5:4300 , piece_id=Z63XTB4CSF365SNHSLPTWPSWWJ7NJ5YH2EG65HZVBANUAU3LLK7Q, seg_pos={0 0}, upload=461212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:33096->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:33096->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33096->45.91.116.5:4300: use of closed network connection Dec 27 13:17:01.633542113 (525h48m1.504625168s) [noise=true , address=45.91.116.5:4300 , piece_id=Z63XTB4CSF365SNHSLPTWPSWWJ7NJ5YH2EG65HZVBANUAU3LLK7Q, seg_pos={0 0}, upload=461212]: Long tail cancellation due to optimal threshold Dec 27 13:17:01.634250391 (525h48m1.505333446s) [noise=true , address=45.91.116.6:4973 , piece_id=AMRZE6G7XDVZIRQ42IEP3IUTVWLX4JID4YVY3UBJFLTYHJY6GYMQ, seg_pos={0 0}, upload=461212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:34800->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 13:17:01.634261443 (525h48m1.505344488s) [noise=true , address=45.91.116.6:4973 , piece_id=AMRZE6G7XDVZIRQ42IEP3IUTVWLX4JID4YVY3UBJFLTYHJY6GYMQ, seg_pos={0 0}, upload=461212]: Long tail cancellation due to optimal threshold Dec 27 13:17:01.634703674 (525h48m1.505786719s) [noise=true , address=77.90.31.17:4770 , piece_id=6KW52BFIVBAJ22EGESQ3KBYNEOF6VCRJJMHOIRKDM5CXM46BKKIA, seg_pos={0 0}, upload=461212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: context canceled; context canceled Dec 27 13:17:01.634715157 (525h48m1.505798202s) [noise=true , address=77.90.31.17:4770 , piece_id=6KW52BFIVBAJ22EGESQ3KBYNEOF6VCRJJMHOIRKDM5CXM46BKKIA, seg_pos={0 0}, upload=461212]: Long tail cancellation due to optimal threshold Dec 27 13:17:01.634741967 (525h48m1.505825012s) [seg_pos={0 0}, upload=461212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:17:01.634870803 (525h48m1.505953848s) [upload=461212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:17:01.634879749 (525h48m1.505962794s) [upload=461212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:17:01.940383913 (525h48m1.811466968s) [upload=461212]: Done uploading segments... Dec 27 13:17:01.940413299 (525h48m1.811496354s) [upload=461212]: Upload finished. err: <nil> Duration: 30.003433743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.............................X...X..] duration: 55.211521443s trace: https://jaeger.dev.storj.io/trace/3fff9d1180db5d69 FAIL (2025-12-27 13:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: 4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ) context deadline exceeded; piecestore: (Node ID: 1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu, Piece ID: 2NBPJXVK52L2M7HNYEH3V5XMA5TJ3O6C27PON23W5QLXYU3S6PAA) context deadline exceeded ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=WYPVV5ND6727W3RUHKVEBW7TY2SLK44OHGIWKNQS3Z7VRA7R6BHA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.313890357 (525h45m29.184973402s) [noise=true , address=77.90.31.19:4874 , piece_id=SXRHPQ6ZKSYS7XMDFM4NM444R3UD77XDCZCLDJGYNVXMC5ZSYWAQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314231116 (525h45m29.185314151s) [noise=true , address=45.91.116.5:4329 , piece_id=4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314397303 (525h45m29.185480338s) [noise=true , address=77.90.31.15:4712 , piece_id=PGSHIFGJWOYKM6E7PWUDTUZP6S32PETNZLD73GL3ZMIZP4REYQMA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314590300 (525h45m29.185673345s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314728854 (525h45m29.185811889s) [noise=true , address=145.79.227.40:4062 , piece_id=PN5ZDTWCSQEQXCRG7Q2KWJ35JA26CIVQCEXIPYH7MSEFCW55JAXA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314946217 (525h45m29.186029262s) [noise=true , address=145.79.227.48:4311 , piece_id=JPY7E33FC6AWCECY6IQEXPMRUSCR6EDJXRFARMU7ZWXS5D7RRVUQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315053011 (525h45m29.186136046s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315309129 (525h45m29.186392174s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315374393 (525h45m29.186457428s) [noise=true , address=45.91.116.4:4964 , piece_id=PS273B6BPO4O6J6G4LQLCVRAOKP67PKW4NZRR5FPN5RPCB2MEQQQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315805594 (525h45m29.186888629s) [noise=true , address=145.79.227.44:4225 , piece_id=45QAKVCQ5FPOLE5DRCG2WSFIOYO7C7NH4HTYHSQILOUFM7EFXO2Q, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316139150 (525h45m29.187222185s) [noise=true , address=77.90.31.21:4894 , piece_id=GR5PRAIVWGPPSNDO2IKB7VN65UBASLHZELBLHORDGYOQA4LWZRSA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316474948 (525h45m29.187557983s) [noise=true , address=145.79.227.30:4768 , piece_id=DQ23KUHNR6GG5RA44UOSKTCAIAQWISDVT547E55LOM5RMHRO6PHQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316597522 (525h45m29.187680627s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316807552 (525h45m29.187890587s) [noise=true , address=145.79.227.17:4998 , piece_id=7GPON22DFOGVWIFAFSKV5JCWRUGVYAWGSKPHOXU7ACH6AG3TEUJA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.317160294 (525h45m29.188243329s) [noise=true , address=77.90.31.12:4613 , piece_id=UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:30.687966688 (525h45m30.559049733s) [upload=461177]: Next returned nil segment Dec 27 13:14:30.687980184 (525h45m30.559063219s) [upload=461177]: Waiting for error group managing segments... Dec 27 13:14:31.040556219 (525h45m30.911639264s) [noise=true , address=193.149.164.143:4701 , piece_id=WYPVV5ND6727W3RUHKVEBW7TY2SLK44OHGIWKNQS3Z7VRA7R6BHA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.181412965 (525h45m31.05249601s) [noise=true , address=145.79.227.48:4311 , piece_id=JPY7E33FC6AWCECY6IQEXPMRUSCR6EDJXRFARMU7ZWXS5D7RRVUQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.185708925 (525h45m31.056791961s) [noise=true , address=145.79.227.40:4062 , piece_id=PN5ZDTWCSQEQXCRG7Q2KWJ35JA26CIVQCEXIPYH7MSEFCW55JAXA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.220843017 (525h45m31.091926062s) [noise=true , address=145.79.227.17:4998 , piece_id=7GPON22DFOGVWIFAFSKV5JCWRUGVYAWGSKPHOXU7ACH6AG3TEUJA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.303221950 (525h45m31.174304995s) [noise=true , address=145.79.227.52:4478 , piece_id=TPSQDF6VYSA7UY4GOJBXN5HX7NGMHKXOS35ICXIUVQCDZYILISBA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.396094786 (525h45m31.267177831s) [noise=true , address=77.90.31.12:4613 , piece_id=UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.467160147 (525h45m31.338243192s) [noise=true , address=77.90.31.14:4687 , piece_id=2NBPJXVK52L2M7HNYEH3V5XMA5TJ3O6C27PON23W5QLXYU3S6PAA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.494978408 (525h45m31.366061453s) [noise=true , address=77.90.31.19:4874 , piece_id=SXRHPQ6ZKSYS7XMDFM4NM444R3UD77XDCZCLDJGYNVXMC5ZSYWAQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.550392295 (525h45m31.42147534s) [noise=true , address=45.91.116.4:4964 , piece_id=PS273B6BPO4O6J6G4LQLCVRAOKP67PKW4NZRR5FPN5RPCB2MEQQQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.731063575 (525h45m31.60214661s) [noise=true , address=45.91.116.5:4329 , piece_id=4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.811147498 (525h45m31.682230543s) [noise=true , address=77.90.31.15:4712 , piece_id=PGSHIFGJWOYKM6E7PWUDTUZP6S32PETNZLD73GL3ZMIZP4REYQMA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.070197935 (525h45m31.94128098s) [noise=true , address=145.79.227.44:4225 , piece_id=45QAKVCQ5FPOLE5DRCG2WSFIOYO7C7NH4HTYHSQILOUFM7EFXO2Q, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.081701990 (525h45m31.952785025s) [noise=true , address=145.79.227.30:4768 , piece_id=DQ23KUHNR6GG5RA44UOSKTCAIAQWISDVT547E55LOM5RMHRO6PHQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.087629396 (525h45m31.958712432s) [noise=true , address=77.90.31.21:4894 , piece_id=GR5PRAIVWGPPSNDO2IKB7VN65UBASLHZELBLHORDGYOQA4LWZRSA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.087648273 (525h45m31.958731318s) [seg_pos={0 0}, upload=461177]: Segment reached optimal threshold of 14 pieces. Dec 27 13:14:32.087908097 (525h45m31.958991142s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection Dec 27 13:14:32.087914629 (525h45m31.958997664s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.087964986 (525h45m31.959048021s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.087980906 (525h45m31.959063941s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088048234 (525h45m31.959131279s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.088053634 (525h45m31.959136669s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088203389 (525h45m31.959286434s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.088208870 (525h45m31.959291905s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088227615 (525h45m31.95931065s) [seg_pos={0 0}, upload=461177]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:14:32.088294693 (525h45m31.959377728s) [upload=461177]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:14:32.088299031 (525h45m31.959382066s) [upload=461177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:14:32.389880444 (525h45m32.260963489s) [upload=461177]: Done uploading segments... Dec 27 13:14:32.389899390 (525h45m32.260982426s) [upload=461177]: Upload finished. err: <nil> Duration: 55.21153591s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)