Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 902277 reports (Page 11312 of 18046)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [........................X...................................]
duration: 
3.597806497s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:52:40 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.597829992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X....X.X.]
duration: 
20.058003614s
trace: 
https://jaeger.dev.storj.io/trace/a11175e06f4cb9d
FAIL (2025-12-29 02:52:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HQBOTTRXC2UJFSPMNB6AXMIWVO52V7EIGFNDKMOLZ3TCMY4ROVMQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 75SB35ASVTUJZXHBGOPGWCJAWCRNHIAHIU5EN2NOVKUYKUO57DQQ) context deadline exceeded
======LOGS======
Dec 29 02:51:02.958104984 (563h22m2.829188029s) [upload=493931]: Starting upload
Dec 29 02:51:02.958271752 (563h22m2.829354787s) [upload=493931]: Done starting upload
Dec 29 02:51:02.958377513 (563h22m2.829460548s) [upload=493931]: Uploading segments...
Dec 29 02:51:02.958603173 (563h22m2.829686218s) [upload=493931]: Got next segment. Inline: false
Dec 29 02:51:02.958653820 (563h22m2.829736875s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:51:02.958659861 (563h22m2.829742926s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:51:03.209540619 (563h22m3.080623674s) [seg_pos={0 0}, upload=493931]: Begin upload segment...
Dec 29 02:51:03.209651290 (563h22m3.080734335s) [seg_pos={0 0}, upload=493931]: Done begin upload segment.
Dec 29 02:51:03.209713328 (563h22m3.080796363s) [noise=true , address=77.90.31.15:4700     , piece_id=HQBOTTRXC2UJFSPMNB6AXMIWVO52V7EIGFNDKMOLZ3TCMY4ROVMQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.209783422 (563h22m3.080866477s) [noise=true , address=45.91.116.11:4120    , piece_id=75SB35ASVTUJZXHBGOPGWCJAWCRNHIAHIU5EN2NOVKUYKUO57DQQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.210559940 (563h22m3.081642995s) [noise=true , address=145.79.227.4:4185    , piece_id=JU4CLC2JX775QPQDJ7RJAYFJY2HJL7OR45G7ZNJLW2IG7OJBYPDA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.210947488 (563h22m3.082030523s) [noise=true , address=45.91.116.13:4156    , piece_id=3CFHHJABZ5EMJI2E4VGTWDNUGPDV35ZBRVVWE73IJ75HXU7GSOVQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.211420168 (563h22m3.082503203s) [noise=true , address=145.79.227.13:4802   , piece_id=356LTFPNPMSF75ZXQQ2UOUKPNQUNU5ZSX24S5HVFKOEEGHBPOU5A, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.211538032 (563h22m3.082621077s) [noise=true , address=45.91.116.4:4940     , piece_id=ZFVTWU34UCGNMKLC6E7H65IWOZTPJJ7VGXOOODGHWYAIAFII3XPQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212018878 (563h22m3.083101923s) [noise=true , address=145.79.227.45:4229   , piece_id=TZINRKC7P6MZDQMHVRCVP3EFAZL3M3QGI6PMVBX4ICVKUVJCRBOA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212047543 (563h22m3.083130578s) [noise=true , address=45.91.116.14:4189    , piece_id=EDFKIQDVX5SVPPBIX3HX7AEDQQRWANWV6BUDH2LUI2KNSRXGR3XA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212550109 (563h22m3.083633144s) [noise=true , address=145.79.227.17:4971   , piece_id=OER7EO46E73OP426GISTCP53CAHI2AOESOZJVXGURVZD7RJC2WDQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212577822 (563h22m3.083660857s) [noise=true , address=145.79.227.26:4653   , piece_id=6TQWLMCQ3WIKJLF75Y4A5NAXMJVRWWRFHK3KATHUHUGXZZESHM3A, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.213130355 (563h22m3.08421339s) [noise=true , address=145.79.227.14:4838   , piece_id=DCVGEOC2PRI2MYSLRKGBI4AR5UB5SBEINQ6UUAFW7BBBRIFUFVDQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.213391602 (563h22m3.084474647s) [noise=true , address=145.79.227.42:4121   , piece_id=WIGDJHQIAIVHNOJXFWKXGSTO7PGXD45N3IILD4NDIHYX4CDJRHLQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.213610638 (563h22m3.084693673s) [noise=true , address=145.79.227.52:4447   , piece_id=UDTIC3KYCR6BVUE5U6OV3R5WPZ6P5EXNUJQC2JBZ3T3KFXXY7TKA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.214074713 (563h22m3.085157758s) [noise=true , address=145.79.227.18:4007   , piece_id=Y3LBNVW6BBQ6ZNVDNWS7JOD44TLTALW2UMJX2FHGK6KR7WVT5XXQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.214095832 (563h22m3.085178867s) [noise=true , address=145.79.227.6:4259    , piece_id=ZHCM7YOOPJF4RHJB4M6OBAR5BTNNSKM5TOM37HZPKKIMAVCJL47Q, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.214629159 (563h22m3.085712204s) [noise=true , address=45.91.116.5:4300     , piece_id=OL25RTQMWZ3MA5NZ4HN6UQGF2UZ6ZCXKFITDIKZ2BDIHROULMABQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:04.833745255 (563h22m4.70482829s) [upload=493931]: Next returned nil segment
Dec 29 02:51:04.833762478 (563h22m4.704845523s) [upload=493931]: Waiting for error group managing segments...
Dec 29 02:51:05.218387150 (563h22m5.089470185s) [noise=true , address=145.79.227.14:4838   , piece_id=DCVGEOC2PRI2MYSLRKGBI4AR5UB5SBEINQ6UUAFW7BBBRIFUFVDQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.220314580 (563h22m5.091397625s) [noise=true , address=145.79.227.4:4185    , piece_id=JU4CLC2JX775QPQDJ7RJAYFJY2HJL7OR45G7ZNJLW2IG7OJBYPDA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.229327252 (563h22m5.100410297s) [noise=true , address=145.79.227.13:4802   , piece_id=356LTFPNPMSF75ZXQQ2UOUKPNQUNU5ZSX24S5HVFKOEEGHBPOU5A, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.229756469 (563h22m5.100839514s) [noise=true , address=145.79.227.26:4653   , piece_id=6TQWLMCQ3WIKJLF75Y4A5NAXMJVRWWRFHK3KATHUHUGXZZESHM3A, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.231045754 (563h22m5.102128799s) [noise=true , address=145.79.227.45:4229   , piece_id=TZINRKC7P6MZDQMHVRCVP3EFAZL3M3QGI6PMVBX4ICVKUVJCRBOA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.236531720 (563h22m5.107614755s) [noise=true , address=145.79.227.17:4971   , piece_id=OER7EO46E73OP426GISTCP53CAHI2AOESOZJVXGURVZD7RJC2WDQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.240808804 (563h22m5.111891839s) [noise=true , address=145.79.227.18:4007   , piece_id=Y3LBNVW6BBQ6ZNVDNWS7JOD44TLTALW2UMJX2FHGK6KR7WVT5XXQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.241644574 (563h22m5.112727609s) [noise=true , address=145.79.227.52:4447   , piece_id=UDTIC3KYCR6BVUE5U6OV3R5WPZ6P5EXNUJQC2JBZ3T3KFXXY7TKA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.254566951 (563h22m5.125649986s) [noise=true , address=145.79.227.42:4121   , piece_id=WIGDJHQIAIVHNOJXFWKXGSTO7PGXD45N3IILD4NDIHYX4CDJRHLQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.498374769 (563h22m5.369457814s) [noise=true , address=45.91.116.4:4940     , piece_id=ZFVTWU34UCGNMKLC6E7H65IWOZTPJJ7VGXOOODGHWYAIAFII3XPQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.546062302 (563h22m5.417145348s) [noise=true , address=45.91.116.13:4156    , piece_id=3CFHHJABZ5EMJI2E4VGTWDNUGPDV35ZBRVVWE73IJ75HXU7GSOVQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.560455719 (563h22m5.431538754s) [noise=true , address=45.91.116.11:4120    , piece_id=75SB35ASVTUJZXHBGOPGWCJAWCRNHIAHIU5EN2NOVKUYKUO57DQQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:06.083619075 (563h22m5.95470212s) [noise=true , address=77.90.31.15:4700     , piece_id=HQBOTTRXC2UJFSPMNB6AXMIWVO52V7EIGFNDKMOLZ3TCMY4ROVMQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:06.083637930 (563h22m5.954720966s) [seg_pos={0 0}, upload=493931]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:51:06.083832191 (563h22m5.954915226s) [noise=true , address=45.91.116.5:4300     , piece_id=OL25RTQMWZ3MA5NZ4HN6UQGF2UZ6ZCXKFITDIKZ2BDIHROULMABQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 29 02:51:06.083838633 (563h22m5.954921678s) [noise=true , address=45.91.116.5:4300     , piece_id=OL25RTQMWZ3MA5NZ4HN6UQGF2UZ6ZCXKFITDIKZ2BDIHROULMABQ, seg_pos={0 0}, upload=493931]: Long tail cancellation due to optimal threshold
Dec 29 02:51:06.083957178 (563h22m5.955040213s) [noise=true , address=145.79.227.6:4259    , piece_id=ZHCM7YOOPJF4RHJB4M6OBAR5BTNNSKM5TOM37HZPKKIMAVCJL47Q, seg_pos={0 0}, upload=493931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:43188->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:43188->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43188->145.79.227.6:4259: use of closed network connection
Dec 29 02:51:06.083962399 (563h22m5.955045434s) [noise=true , address=145.79.227.6:4259    , piece_id=ZHCM7YOOPJF4RHJB4M6OBAR5BTNNSKM5TOM37HZPKKIMAVCJL47Q, seg_pos={0 0}, upload=493931]: Long tail cancellation due to optimal threshold
Dec 29 02:51:06.084225740 (563h22m5.955308785s) [noise=true , address=45.91.116.14:4189    , piece_id=EDFKIQDVX5SVPPBIX3HX7AEDQQRWANWV6BUDH2LUI2KNSRXGR3XA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 29 02:51:06.084245909 (563h22m5.955328954s) [noise=true , address=45.91.116.14:4189    , piece_id=EDFKIQDVX5SVPPBIX3HX7AEDQQRWANWV6BUDH2LUI2KNSRXGR3XA, seg_pos={0 0}, upload=493931]: Long tail cancellation due to optimal threshold
Dec 29 02:51:06.084272809 (563h22m5.955355854s) [seg_pos={0 0}, upload=493931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:51:06.084438655 (563h22m5.9555217s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:51:06.084446249 (563h22m5.955529284s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:51:06.383885508 (563h22m6.254968543s) [upload=493931]: Done uploading segments...
Dec 29 02:51:06.383902981 (563h22m6.254986026s) [upload=493931]: Upload finished. err: <nil>
Duration: 20.058021818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.............................................X..............]
duration: 
18.458148354s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:52:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 18.458190805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........................X..................................]
duration: 
3.593722534s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:51:47 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.593747843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [..X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X....X.X]
duration: 
30.003083455s
trace: 
https://jaeger.dev.storj.io/trace/a11175e06f4cb9d
FAIL (2025-12-29 02:51:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HQBOTTRXC2UJFSPMNB6AXMIWVO52V7EIGFNDKMOLZ3TCMY4ROVMQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 75SB35ASVTUJZXHBGOPGWCJAWCRNHIAHIU5EN2NOVKUYKUO57DQQ) context deadline exceeded
======LOGS======
Dec 29 02:51:02.958104984 (563h22m2.829188029s) [upload=493931]: Starting upload
Dec 29 02:51:02.958271752 (563h22m2.829354787s) [upload=493931]: Done starting upload
Dec 29 02:51:02.958377513 (563h22m2.829460548s) [upload=493931]: Uploading segments...
Dec 29 02:51:02.958603173 (563h22m2.829686218s) [upload=493931]: Got next segment. Inline: false
Dec 29 02:51:02.958653820 (563h22m2.829736875s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:51:02.958659861 (563h22m2.829742926s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:51:03.209540619 (563h22m3.080623674s) [seg_pos={0 0}, upload=493931]: Begin upload segment...
Dec 29 02:51:03.209651290 (563h22m3.080734335s) [seg_pos={0 0}, upload=493931]: Done begin upload segment.
Dec 29 02:51:03.209713328 (563h22m3.080796363s) [noise=true , address=77.90.31.15:4700     , piece_id=HQBOTTRXC2UJFSPMNB6AXMIWVO52V7EIGFNDKMOLZ3TCMY4ROVMQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.209783422 (563h22m3.080866477s) [noise=true , address=45.91.116.11:4120    , piece_id=75SB35ASVTUJZXHBGOPGWCJAWCRNHIAHIU5EN2NOVKUYKUO57DQQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.210559940 (563h22m3.081642995s) [noise=true , address=145.79.227.4:4185    , piece_id=JU4CLC2JX775QPQDJ7RJAYFJY2HJL7OR45G7ZNJLW2IG7OJBYPDA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.210947488 (563h22m3.082030523s) [noise=true , address=45.91.116.13:4156    , piece_id=3CFHHJABZ5EMJI2E4VGTWDNUGPDV35ZBRVVWE73IJ75HXU7GSOVQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.211420168 (563h22m3.082503203s) [noise=true , address=145.79.227.13:4802   , piece_id=356LTFPNPMSF75ZXQQ2UOUKPNQUNU5ZSX24S5HVFKOEEGHBPOU5A, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.211538032 (563h22m3.082621077s) [noise=true , address=45.91.116.4:4940     , piece_id=ZFVTWU34UCGNMKLC6E7H65IWOZTPJJ7VGXOOODGHWYAIAFII3XPQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212018878 (563h22m3.083101923s) [noise=true , address=145.79.227.45:4229   , piece_id=TZINRKC7P6MZDQMHVRCVP3EFAZL3M3QGI6PMVBX4ICVKUVJCRBOA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212047543 (563h22m3.083130578s) [noise=true , address=45.91.116.14:4189    , piece_id=EDFKIQDVX5SVPPBIX3HX7AEDQQRWANWV6BUDH2LUI2KNSRXGR3XA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212550109 (563h22m3.083633144s) [noise=true , address=145.79.227.17:4971   , piece_id=OER7EO46E73OP426GISTCP53CAHI2AOESOZJVXGURVZD7RJC2WDQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.212577822 (563h22m3.083660857s) [noise=true , address=145.79.227.26:4653   , piece_id=6TQWLMCQ3WIKJLF75Y4A5NAXMJVRWWRFHK3KATHUHUGXZZESHM3A, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.213130355 (563h22m3.08421339s) [noise=true , address=145.79.227.14:4838   , piece_id=DCVGEOC2PRI2MYSLRKGBI4AR5UB5SBEINQ6UUAFW7BBBRIFUFVDQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.213391602 (563h22m3.084474647s) [noise=true , address=145.79.227.42:4121   , piece_id=WIGDJHQIAIVHNOJXFWKXGSTO7PGXD45N3IILD4NDIHYX4CDJRHLQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.213610638 (563h22m3.084693673s) [noise=true , address=145.79.227.52:4447   , piece_id=UDTIC3KYCR6BVUE5U6OV3R5WPZ6P5EXNUJQC2JBZ3T3KFXXY7TKA, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.214074713 (563h22m3.085157758s) [noise=true , address=145.79.227.18:4007   , piece_id=Y3LBNVW6BBQ6ZNVDNWS7JOD44TLTALW2UMJX2FHGK6KR7WVT5XXQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.214095832 (563h22m3.085178867s) [noise=true , address=145.79.227.6:4259    , piece_id=ZHCM7YOOPJF4RHJB4M6OBAR5BTNNSKM5TOM37HZPKKIMAVCJL47Q, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:03.214629159 (563h22m3.085712204s) [noise=true , address=45.91.116.5:4300     , piece_id=OL25RTQMWZ3MA5NZ4HN6UQGF2UZ6ZCXKFITDIKZ2BDIHROULMABQ, seg_pos={0 0}, upload=493931]: Uploading piece...
Dec 29 02:51:04.833745255 (563h22m4.70482829s) [upload=493931]: Next returned nil segment
Dec 29 02:51:04.833762478 (563h22m4.704845523s) [upload=493931]: Waiting for error group managing segments...
Dec 29 02:51:05.218387150 (563h22m5.089470185s) [noise=true , address=145.79.227.14:4838   , piece_id=DCVGEOC2PRI2MYSLRKGBI4AR5UB5SBEINQ6UUAFW7BBBRIFUFVDQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.220314580 (563h22m5.091397625s) [noise=true , address=145.79.227.4:4185    , piece_id=JU4CLC2JX775QPQDJ7RJAYFJY2HJL7OR45G7ZNJLW2IG7OJBYPDA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.229327252 (563h22m5.100410297s) [noise=true , address=145.79.227.13:4802   , piece_id=356LTFPNPMSF75ZXQQ2UOUKPNQUNU5ZSX24S5HVFKOEEGHBPOU5A, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.229756469 (563h22m5.100839514s) [noise=true , address=145.79.227.26:4653   , piece_id=6TQWLMCQ3WIKJLF75Y4A5NAXMJVRWWRFHK3KATHUHUGXZZESHM3A, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.231045754 (563h22m5.102128799s) [noise=true , address=145.79.227.45:4229   , piece_id=TZINRKC7P6MZDQMHVRCVP3EFAZL3M3QGI6PMVBX4ICVKUVJCRBOA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.236531720 (563h22m5.107614755s) [noise=true , address=145.79.227.17:4971   , piece_id=OER7EO46E73OP426GISTCP53CAHI2AOESOZJVXGURVZD7RJC2WDQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.240808804 (563h22m5.111891839s) [noise=true , address=145.79.227.18:4007   , piece_id=Y3LBNVW6BBQ6ZNVDNWS7JOD44TLTALW2UMJX2FHGK6KR7WVT5XXQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.241644574 (563h22m5.112727609s) [noise=true , address=145.79.227.52:4447   , piece_id=UDTIC3KYCR6BVUE5U6OV3R5WPZ6P5EXNUJQC2JBZ3T3KFXXY7TKA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.254566951 (563h22m5.125649986s) [noise=true , address=145.79.227.42:4121   , piece_id=WIGDJHQIAIVHNOJXFWKXGSTO7PGXD45N3IILD4NDIHYX4CDJRHLQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.498374769 (563h22m5.369457814s) [noise=true , address=45.91.116.4:4940     , piece_id=ZFVTWU34UCGNMKLC6E7H65IWOZTPJJ7VGXOOODGHWYAIAFII3XPQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.546062302 (563h22m5.417145348s) [noise=true , address=45.91.116.13:4156    , piece_id=3CFHHJABZ5EMJI2E4VGTWDNUGPDV35ZBRVVWE73IJ75HXU7GSOVQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:05.560455719 (563h22m5.431538754s) [noise=true , address=45.91.116.11:4120    , piece_id=75SB35ASVTUJZXHBGOPGWCJAWCRNHIAHIU5EN2NOVKUYKUO57DQQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:06.083619075 (563h22m5.95470212s) [noise=true , address=77.90.31.15:4700     , piece_id=HQBOTTRXC2UJFSPMNB6AXMIWVO52V7EIGFNDKMOLZ3TCMY4ROVMQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: <nil>
Dec 29 02:51:06.083637930 (563h22m5.954720966s) [seg_pos={0 0}, upload=493931]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:51:06.083832191 (563h22m5.954915226s) [noise=true , address=45.91.116.5:4300     , piece_id=OL25RTQMWZ3MA5NZ4HN6UQGF2UZ6ZCXKFITDIKZ2BDIHROULMABQ, seg_pos={0 0}, upload=493931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 29 02:51:06.083838633 (563h22m5.954921678s) [noise=true , address=45.91.116.5:4300     , piece_id=OL25RTQMWZ3MA5NZ4HN6UQGF2UZ6ZCXKFITDIKZ2BDIHROULMABQ, seg_pos={0 0}, upload=493931]: Long tail cancellation due to optimal threshold
Dec 29 02:51:06.083957178 (563h22m5.955040213s) [noise=true , address=145.79.227.6:4259    , piece_id=ZHCM7YOOPJF4RHJB4M6OBAR5BTNNSKM5TOM37HZPKKIMAVCJL47Q, seg_pos={0 0}, upload=493931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:43188->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:43188->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43188->145.79.227.6:4259: use of closed network connection
Dec 29 02:51:06.083962399 (563h22m5.955045434s) [noise=true , address=145.79.227.6:4259    , piece_id=ZHCM7YOOPJF4RHJB4M6OBAR5BTNNSKM5TOM37HZPKKIMAVCJL47Q, seg_pos={0 0}, upload=493931]: Long tail cancellation due to optimal threshold
Dec 29 02:51:06.084225740 (563h22m5.955308785s) [noise=true , address=45.91.116.14:4189    , piece_id=EDFKIQDVX5SVPPBIX3HX7AEDQQRWANWV6BUDH2LUI2KNSRXGR3XA, seg_pos={0 0}, upload=493931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 29 02:51:06.084245909 (563h22m5.955328954s) [noise=true , address=45.91.116.14:4189    , piece_id=EDFKIQDVX5SVPPBIX3HX7AEDQQRWANWV6BUDH2LUI2KNSRXGR3XA, seg_pos={0 0}, upload=493931]: Long tail cancellation due to optimal threshold
Dec 29 02:51:06.084272809 (563h22m5.955355854s) [seg_pos={0 0}, upload=493931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:51:06.084438655 (563h22m5.9555217s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:51:06.084446249 (563h22m5.955529284s) [upload=493931]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:51:06.383885508 (563h22m6.254968543s) [upload=493931]: Done uploading segments...
Dec 29 02:51:06.383902981 (563h22m6.254986026s) [upload=493931]: Upload finished. err: <nil>
Duration: 30.003105106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..............................................X.............]
duration: 
38.336592913s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:51:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 38.336609285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..........................X.................................]
duration: 
3.44834254s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:50:51 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.448370533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [...X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X....X.]
duration: 
16.080214521s
trace: 
https://jaeger.dev.storj.io/trace/3bcb00971bb83c8e
FAIL (2025-12-29 02:50:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: HBNXVAEVO4E3FL22OFO7NZE4GTQKXSWDEGSWMZCNHOHBSQ2DWOXQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: PRIPKHPP7K462C6MWKFNTFTOYI36KJBXFSJGMN6JEUGBBO6LFUUA) context deadline exceeded
======LOGS======
ec 29 02:49:43.467567206 (563h20m43.338650251s) [upload=493911]: Starting upload
Dec 29 02:49:43.467668008 (563h20m43.338751053s) [upload=493911]: Done starting upload
Dec 29 02:49:43.467933814 (563h20m43.339016869s) [upload=493911]: Uploading segments...
Dec 29 02:49:43.467993929 (563h20m43.339076974s) [upload=493911]: Got next segment. Inline: false
Dec 29 02:49:43.468055566 (563h20m43.339138611s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:49:43.468062589 (563h20m43.339145634s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:49:43.710703507 (563h20m43.581786542s) [seg_pos={0 0}, upload=493911]: Begin upload segment...
Dec 29 02:49:43.710749194 (563h20m43.581832229s) [seg_pos={0 0}, upload=493911]: Done begin upload segment.
Dec 29 02:49:43.710776887 (563h20m43.581859922s) [noise=true , address=45.91.116.7:4010     , piece_id=HBNXVAEVO4E3FL22OFO7NZE4GTQKXSWDEGSWMZCNHOHBSQ2DWOXQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.711027875 (563h20m43.58211091s) [noise=true , address=77.90.31.12:4590     , piece_id=74IKYB2PY6ITZCWLKGQYRQQKKCP6XLPQTAEJGYRI7FGUCZBJ3OCA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.711630302 (563h20m43.582713337s) [noise=true , address=45.91.116.12:4227    , piece_id=MTMXSGYX3MYMFXJKNIXHZWCEXDFKIKRYWVD6CX7YRNFA3GMGEQRA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.712048559 (563h20m43.583131594s) [noise=true , address=145.79.227.32:4834   , piece_id=WPVFYPEMRVTBWDSF36CRJP6JYY2IV44HFAALS3FJZFES4YUDCLDQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.712457998 (563h20m43.583541043s) [noise=true , address=145.79.227.52:4445   , piece_id=G6J2VCLKQXKE4QYIGZCB6U55OQMLZP6W2LTUQZY3L5MBFUD4GNGQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.712914256 (563h20m43.583997301s) [noise=true , address=145.79.227.48:4303   , piece_id=PYO6BP7WAD7ZYM4TVIPB42FQS6VFM7VJCXPIK6TCLKFOJOR5FAWQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.713368712 (563h20m43.584451747s) [noise=true , address=145.79.227.47:4266   , piece_id=WKULCRGISAJ2P7WFBMLBJFBPNJOZMOMIISLLHNHRP256KHU46AGA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.714399395 (563h20m43.58548243s) [noise=true , address=45.91.116.6:4976     , piece_id=X47UFTX4NTXOKM3LM4R46P7A7B5UA22B64FF6VYMCACZSXANQIFQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.714939322 (563h20m43.586022367s) [noise=true , address=77.90.31.21:4880     , piece_id=PRIPKHPP7K462C6MWKFNTFTOYI36KJBXFSJGMN6JEUGBBO6LFUUA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.715588228 (563h20m43.586671263s) [noise=true , address=145.79.227.45:4229   , piece_id=WAEZMCAUUZ2QJMZRUSF5XOLMEOU4OW5EIOVPFIFW6UOPOYWJOIDQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.716580547 (563h20m43.587663592s) [noise=true , address=145.79.227.38:4979   , piece_id=DJSBXR4FDFCXICLWYQXOZUGHJCTNDAVXXEO4D25JWBW4DOFXYFJQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.716680317 (563h20m43.587763352s) [noise=true , address=145.79.227.42:4123   , piece_id=BSPKNLHTP6VSPJLKSZYP3I77UITUUQOQOUSTTXTUV2C2BOFYJAIQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.717877066 (563h20m43.588960111s) [noise=true , address=145.79.227.51:4410   , piece_id=L7L3UWEJWY4NHXMVPNWGKKAKAM6HVETMQIY3MBVQVZG7AISCRTFQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.719056822 (563h20m43.590139857s) [noise=true , address=145.79.227.33:4869   , piece_id=K4ZURAXW7K5DAMI6W6EIPIRUB6TBKVPI5X7JA5GAVOQ5VT66JGWQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.719705728 (563h20m43.590788773s) [noise=true , address=145.79.227.12:4765   , piece_id=ZPDG4VY35VSQNUIYK3MANHH35KIAAKEZGM5TAGJK377K4HFB3BYQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.721539009 (563h20m43.592622054s) [noise=true , address=77.90.31.10:4520     , piece_id=F3BCQASKROB4AZCVROKQRFM4QJDNFY2IJPW2D7HOH4U4NDFXCF3A, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:45.336460345 (563h20m45.20754338s) [upload=493911]: Next returned nil segment
Dec 29 02:49:45.336473219 (563h20m45.207556255s) [upload=493911]: Waiting for error group managing segments...
Dec 29 02:49:45.718953549 (563h20m45.590036594s) [noise=true , address=145.79.227.52:4445   , piece_id=G6J2VCLKQXKE4QYIGZCB6U55OQMLZP6W2LTUQZY3L5MBFUD4GNGQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.722255116 (563h20m45.593338151s) [noise=true , address=145.79.227.48:4303   , piece_id=PYO6BP7WAD7ZYM4TVIPB42FQS6VFM7VJCXPIK6TCLKFOJOR5FAWQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.725910516 (563h20m45.596993561s) [noise=true , address=145.79.227.12:4765   , piece_id=ZPDG4VY35VSQNUIYK3MANHH35KIAAKEZGM5TAGJK377K4HFB3BYQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.733262115 (563h20m45.60434515s) [noise=true , address=145.79.227.45:4229   , piece_id=WAEZMCAUUZ2QJMZRUSF5XOLMEOU4OW5EIOVPFIFW6UOPOYWJOIDQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.733978920 (563h20m45.605061955s) [noise=true , address=145.79.227.51:4410   , piece_id=L7L3UWEJWY4NHXMVPNWGKKAKAM6HVETMQIY3MBVQVZG7AISCRTFQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.734134646 (563h20m45.605217691s) [noise=true , address=145.79.227.32:4834   , piece_id=WPVFYPEMRVTBWDSF36CRJP6JYY2IV44HFAALS3FJZFES4YUDCLDQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.741957431 (563h20m45.613040476s) [noise=true , address=145.79.227.42:4123   , piece_id=BSPKNLHTP6VSPJLKSZYP3I77UITUUQOQOUSTTXTUV2C2BOFYJAIQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.743221207 (563h20m45.614304252s) [noise=true , address=145.79.227.38:4979   , piece_id=DJSBXR4FDFCXICLWYQXOZUGHJCTNDAVXXEO4D25JWBW4DOFXYFJQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.745627770 (563h20m45.616710815s) [noise=true , address=145.79.227.47:4266   , piece_id=WKULCRGISAJ2P7WFBMLBJFBPNJOZMOMIISLLHNHRP256KHU46AGA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.964359222 (563h20m45.835442257s) [noise=true , address=77.90.31.21:4880     , piece_id=PRIPKHPP7K462C6MWKFNTFTOYI36KJBXFSJGMN6JEUGBBO6LFUUA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.966364810 (563h20m45.837447845s) [noise=true , address=45.91.116.7:4010     , piece_id=HBNXVAEVO4E3FL22OFO7NZE4GTQKXSWDEGSWMZCNHOHBSQ2DWOXQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:46.008413035 (563h20m45.87949608s) [noise=true , address=77.90.31.10:4520     , piece_id=F3BCQASKROB4AZCVROKQRFM4QJDNFY2IJPW2D7HOH4U4NDFXCF3A, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:46.068912864 (563h20m45.939995899s) [noise=true , address=45.91.116.12:4227    , piece_id=MTMXSGYX3MYMFXJKNIXHZWCEXDFKIKRYWVD6CX7YRNFA3GMGEQRA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:46.068932653 (563h20m45.940015688s) [seg_pos={0 0}, upload=493911]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:49:46.069506916 (563h20m45.940589951s) [noise=true , address=145.79.227.33:4869   , piece_id=K4ZURAXW7K5DAMI6W6EIPIRUB6TBKVPI5X7JA5GAVOQ5VT66JGWQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 02:49:46.069519579 (563h20m45.940602614s) [noise=true , address=145.79.227.33:4869   , piece_id=K4ZURAXW7K5DAMI6W6EIPIRUB6TBKVPI5X7JA5GAVOQ5VT66JGWQ, seg_pos={0 0}, upload=493911]: Long tail cancellation due to optimal threshold
Dec 29 02:49:46.069760558 (563h20m45.940843593s) [noise=true , address=77.90.31.12:4590     , piece_id=74IKYB2PY6ITZCWLKGQYRQQKKCP6XLPQTAEJGYRI7FGUCZBJ3OCA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:44422->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:44422->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->77.90.31.12:4590: use of closed network connection
Dec 29 02:49:46.069782009 (563h20m45.940865045s) [noise=true , address=77.90.31.12:4590     , piece_id=74IKYB2PY6ITZCWLKGQYRQQKKCP6XLPQTAEJGYRI7FGUCZBJ3OCA, seg_pos={0 0}, upload=493911]: Long tail cancellation due to optimal threshold
Dec 29 02:49:46.070045130 (563h20m45.941128165s) [noise=true , address=45.91.116.6:4976     , piece_id=X47UFTX4NTXOKM3LM4R46P7A7B5UA22B64FF6VYMCACZSXANQIFQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 29 02:49:46.070057294 (563h20m45.941140329s) [noise=true , address=45.91.116.6:4976     , piece_id=X47UFTX4NTXOKM3LM4R46P7A7B5UA22B64FF6VYMCACZSXANQIFQ, seg_pos={0 0}, upload=493911]: Long tail cancellation due to optimal threshold
Dec 29 02:49:46.070151043 (563h20m45.941234078s) [seg_pos={0 0}, upload=493911]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:49:46.070354910 (563h20m45.941437945s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:49:46.070362255 (563h20m45.94144529s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:49:46.364992908 (563h20m46.236075953s) [upload=493911]: Done uploading segments...
Dec 29 02:49:46.365012464 (563h20m46.236095499s) [upload=493911]: Upload finished. err: <nil>
Duration: 16.080243497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...............................................X............]
duration: 
43.59371232s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:50:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 43.593749341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X....X]
duration: 
30.002632932s
trace: 
https://jaeger.dev.storj.io/trace/3bcb00971bb83c8e
FAIL (2025-12-29 02:50:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: HBNXVAEVO4E3FL22OFO7NZE4GTQKXSWDEGSWMZCNHOHBSQ2DWOXQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: PRIPKHPP7K462C6MWKFNTFTOYI36KJBXFSJGMN6JEUGBBO6LFUUA) context deadline exceeded
======LOGS======
ec 29 02:49:43.467567206 (563h20m43.338650251s) [upload=493911]: Starting upload
Dec 29 02:49:43.467668008 (563h20m43.338751053s) [upload=493911]: Done starting upload
Dec 29 02:49:43.467933814 (563h20m43.339016869s) [upload=493911]: Uploading segments...
Dec 29 02:49:43.467993929 (563h20m43.339076974s) [upload=493911]: Got next segment. Inline: false
Dec 29 02:49:43.468055566 (563h20m43.339138611s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:49:43.468062589 (563h20m43.339145634s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:49:43.710703507 (563h20m43.581786542s) [seg_pos={0 0}, upload=493911]: Begin upload segment...
Dec 29 02:49:43.710749194 (563h20m43.581832229s) [seg_pos={0 0}, upload=493911]: Done begin upload segment.
Dec 29 02:49:43.710776887 (563h20m43.581859922s) [noise=true , address=45.91.116.7:4010     , piece_id=HBNXVAEVO4E3FL22OFO7NZE4GTQKXSWDEGSWMZCNHOHBSQ2DWOXQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.711027875 (563h20m43.58211091s) [noise=true , address=77.90.31.12:4590     , piece_id=74IKYB2PY6ITZCWLKGQYRQQKKCP6XLPQTAEJGYRI7FGUCZBJ3OCA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.711630302 (563h20m43.582713337s) [noise=true , address=45.91.116.12:4227    , piece_id=MTMXSGYX3MYMFXJKNIXHZWCEXDFKIKRYWVD6CX7YRNFA3GMGEQRA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.712048559 (563h20m43.583131594s) [noise=true , address=145.79.227.32:4834   , piece_id=WPVFYPEMRVTBWDSF36CRJP6JYY2IV44HFAALS3FJZFES4YUDCLDQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.712457998 (563h20m43.583541043s) [noise=true , address=145.79.227.52:4445   , piece_id=G6J2VCLKQXKE4QYIGZCB6U55OQMLZP6W2LTUQZY3L5MBFUD4GNGQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.712914256 (563h20m43.583997301s) [noise=true , address=145.79.227.48:4303   , piece_id=PYO6BP7WAD7ZYM4TVIPB42FQS6VFM7VJCXPIK6TCLKFOJOR5FAWQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.713368712 (563h20m43.584451747s) [noise=true , address=145.79.227.47:4266   , piece_id=WKULCRGISAJ2P7WFBMLBJFBPNJOZMOMIISLLHNHRP256KHU46AGA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.714399395 (563h20m43.58548243s) [noise=true , address=45.91.116.6:4976     , piece_id=X47UFTX4NTXOKM3LM4R46P7A7B5UA22B64FF6VYMCACZSXANQIFQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.714939322 (563h20m43.586022367s) [noise=true , address=77.90.31.21:4880     , piece_id=PRIPKHPP7K462C6MWKFNTFTOYI36KJBXFSJGMN6JEUGBBO6LFUUA, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.715588228 (563h20m43.586671263s) [noise=true , address=145.79.227.45:4229   , piece_id=WAEZMCAUUZ2QJMZRUSF5XOLMEOU4OW5EIOVPFIFW6UOPOYWJOIDQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.716580547 (563h20m43.587663592s) [noise=true , address=145.79.227.38:4979   , piece_id=DJSBXR4FDFCXICLWYQXOZUGHJCTNDAVXXEO4D25JWBW4DOFXYFJQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.716680317 (563h20m43.587763352s) [noise=true , address=145.79.227.42:4123   , piece_id=BSPKNLHTP6VSPJLKSZYP3I77UITUUQOQOUSTTXTUV2C2BOFYJAIQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.717877066 (563h20m43.588960111s) [noise=true , address=145.79.227.51:4410   , piece_id=L7L3UWEJWY4NHXMVPNWGKKAKAM6HVETMQIY3MBVQVZG7AISCRTFQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.719056822 (563h20m43.590139857s) [noise=true , address=145.79.227.33:4869   , piece_id=K4ZURAXW7K5DAMI6W6EIPIRUB6TBKVPI5X7JA5GAVOQ5VT66JGWQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.719705728 (563h20m43.590788773s) [noise=true , address=145.79.227.12:4765   , piece_id=ZPDG4VY35VSQNUIYK3MANHH35KIAAKEZGM5TAGJK377K4HFB3BYQ, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:43.721539009 (563h20m43.592622054s) [noise=true , address=77.90.31.10:4520     , piece_id=F3BCQASKROB4AZCVROKQRFM4QJDNFY2IJPW2D7HOH4U4NDFXCF3A, seg_pos={0 0}, upload=493911]: Uploading piece...
Dec 29 02:49:45.336460345 (563h20m45.20754338s) [upload=493911]: Next returned nil segment
Dec 29 02:49:45.336473219 (563h20m45.207556255s) [upload=493911]: Waiting for error group managing segments...
Dec 29 02:49:45.718953549 (563h20m45.590036594s) [noise=true , address=145.79.227.52:4445   , piece_id=G6J2VCLKQXKE4QYIGZCB6U55OQMLZP6W2LTUQZY3L5MBFUD4GNGQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.722255116 (563h20m45.593338151s) [noise=true , address=145.79.227.48:4303   , piece_id=PYO6BP7WAD7ZYM4TVIPB42FQS6VFM7VJCXPIK6TCLKFOJOR5FAWQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.725910516 (563h20m45.596993561s) [noise=true , address=145.79.227.12:4765   , piece_id=ZPDG4VY35VSQNUIYK3MANHH35KIAAKEZGM5TAGJK377K4HFB3BYQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.733262115 (563h20m45.60434515s) [noise=true , address=145.79.227.45:4229   , piece_id=WAEZMCAUUZ2QJMZRUSF5XOLMEOU4OW5EIOVPFIFW6UOPOYWJOIDQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.733978920 (563h20m45.605061955s) [noise=true , address=145.79.227.51:4410   , piece_id=L7L3UWEJWY4NHXMVPNWGKKAKAM6HVETMQIY3MBVQVZG7AISCRTFQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.734134646 (563h20m45.605217691s) [noise=true , address=145.79.227.32:4834   , piece_id=WPVFYPEMRVTBWDSF36CRJP6JYY2IV44HFAALS3FJZFES4YUDCLDQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.741957431 (563h20m45.613040476s) [noise=true , address=145.79.227.42:4123   , piece_id=BSPKNLHTP6VSPJLKSZYP3I77UITUUQOQOUSTTXTUV2C2BOFYJAIQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.743221207 (563h20m45.614304252s) [noise=true , address=145.79.227.38:4979   , piece_id=DJSBXR4FDFCXICLWYQXOZUGHJCTNDAVXXEO4D25JWBW4DOFXYFJQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.745627770 (563h20m45.616710815s) [noise=true , address=145.79.227.47:4266   , piece_id=WKULCRGISAJ2P7WFBMLBJFBPNJOZMOMIISLLHNHRP256KHU46AGA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.964359222 (563h20m45.835442257s) [noise=true , address=77.90.31.21:4880     , piece_id=PRIPKHPP7K462C6MWKFNTFTOYI36KJBXFSJGMN6JEUGBBO6LFUUA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:45.966364810 (563h20m45.837447845s) [noise=true , address=45.91.116.7:4010     , piece_id=HBNXVAEVO4E3FL22OFO7NZE4GTQKXSWDEGSWMZCNHOHBSQ2DWOXQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:46.008413035 (563h20m45.87949608s) [noise=true , address=77.90.31.10:4520     , piece_id=F3BCQASKROB4AZCVROKQRFM4QJDNFY2IJPW2D7HOH4U4NDFXCF3A, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:46.068912864 (563h20m45.939995899s) [noise=true , address=45.91.116.12:4227    , piece_id=MTMXSGYX3MYMFXJKNIXHZWCEXDFKIKRYWVD6CX7YRNFA3GMGEQRA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: <nil>
Dec 29 02:49:46.068932653 (563h20m45.940015688s) [seg_pos={0 0}, upload=493911]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:49:46.069506916 (563h20m45.940589951s) [noise=true , address=145.79.227.33:4869   , piece_id=K4ZURAXW7K5DAMI6W6EIPIRUB6TBKVPI5X7JA5GAVOQ5VT66JGWQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 02:49:46.069519579 (563h20m45.940602614s) [noise=true , address=145.79.227.33:4869   , piece_id=K4ZURAXW7K5DAMI6W6EIPIRUB6TBKVPI5X7JA5GAVOQ5VT66JGWQ, seg_pos={0 0}, upload=493911]: Long tail cancellation due to optimal threshold
Dec 29 02:49:46.069760558 (563h20m45.940843593s) [noise=true , address=77.90.31.12:4590     , piece_id=74IKYB2PY6ITZCWLKGQYRQQKKCP6XLPQTAEJGYRI7FGUCZBJ3OCA, seg_pos={0 0}, upload=493911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:44422->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:44422->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->77.90.31.12:4590: use of closed network connection
Dec 29 02:49:46.069782009 (563h20m45.940865045s) [noise=true , address=77.90.31.12:4590     , piece_id=74IKYB2PY6ITZCWLKGQYRQQKKCP6XLPQTAEJGYRI7FGUCZBJ3OCA, seg_pos={0 0}, upload=493911]: Long tail cancellation due to optimal threshold
Dec 29 02:49:46.070045130 (563h20m45.941128165s) [noise=true , address=45.91.116.6:4976     , piece_id=X47UFTX4NTXOKM3LM4R46P7A7B5UA22B64FF6VYMCACZSXANQIFQ, seg_pos={0 0}, upload=493911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 29 02:49:46.070057294 (563h20m45.941140329s) [noise=true , address=45.91.116.6:4976     , piece_id=X47UFTX4NTXOKM3LM4R46P7A7B5UA22B64FF6VYMCACZSXANQIFQ, seg_pos={0 0}, upload=493911]: Long tail cancellation due to optimal threshold
Dec 29 02:49:46.070151043 (563h20m45.941234078s) [seg_pos={0 0}, upload=493911]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:49:46.070354910 (563h20m45.941437945s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:49:46.070362255 (563h20m45.94144529s) [upload=493911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:49:46.364992908 (563h20m46.236075953s) [upload=493911]: Done uploading segments...
Dec 29 02:49:46.365012464 (563h20m46.236095499s) [upload=493911]: Upload finished. err: <nil>
Duration: 30.002654964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [................................................X...........]
duration: 
28.637848666s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:49:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 28.637867761s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...........................X................................]
duration: 
3.517527784s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:49:37 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.517580135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.................................................X..........]
duration: 
42.237243614s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:49:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 42.237271366s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [............................X...............................]
duration: 
3.623526807s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:49:01 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.623567994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X....]
duration: 
11.270280338s
trace: 
https://jaeger.dev.storj.io/trace/70ac13beb974b2b6
FAIL (2025-12-29 02:48:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A) context deadline exceeded
======LOGS======
ents...
Dec 29 02:44:43.889083534 (563h15m43.760166569s) [upload=493833]: Got next segment. Inline: false
Dec 29 02:44:43.889176521 (563h15m43.760259556s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:44:43.889184266 (563h15m43.760267301s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:44:44.135574794 (563h15m44.006657829s) [seg_pos={0 0}, upload=493833]: Begin upload segment...
Dec 29 02:44:44.135645478 (563h15m44.006728513s) [seg_pos={0 0}, upload=493833]: Done begin upload segment.
Dec 29 02:44:44.135697757 (563h15m44.006780802s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.135704410 (563h15m44.006787445s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136241052 (563h15m44.007324097s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136430152 (563h15m44.007513197s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.137224605 (563h15m44.00830765s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138265126 (563h15m44.009348161s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138784245 (563h15m44.00986728s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.141113890 (563h15m44.012196925s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.142571886 (563h15m44.013654921s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.143980559 (563h15m44.015063594s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.144745244 (563h15m44.015828289s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.146013931 (563h15m44.017096966s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.147269792 (563h15m44.018352827s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148364226 (563h15m44.019447261s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148841184 (563h15m44.019924249s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.150128576 (563h15m44.021211611s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:45.851835745 (563h15m45.72291878s) [upload=493833]: Next returned nil segment
Dec 29 02:44:45.851851295 (563h15m45.72293434s) [upload=493833]: Waiting for error group managing segments...
Dec 29 02:44:46.139870389 (563h15m46.010953434s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.144675569 (563h15m46.015758604s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.149426535 (563h15m46.02050957s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.150548371 (563h15m46.021631406s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155786034 (563h15m46.026869069s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155850717 (563h15m46.026933752s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.156982112 (563h15m46.028065147s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.157752429 (563h15m46.028835474s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162416520 (563h15m46.033499565s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162587144 (563h15m46.033670189s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.181878848 (563h15m46.052961883s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.369052459 (563h15m46.240135514s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445436664 (563h15m46.316519709s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445459768 (563h15m46.316542803s) [seg_pos={0 0}, upload=493833]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:44:46.445738268 (563h15m46.316821314s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection
Dec 29 02:44:46.445748708 (563h15m46.316831753s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.445882603 (563h15m46.316965638s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 29 02:44:46.445917189 (563h15m46.317000224s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446105768 (563h15m46.317188813s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection
Dec 29 02:44:46.446112530 (563h15m46.317195575s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446142588 (563h15m46.317225623s) [seg_pos={0 0}, upload=493833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:44:46.446260082 (563h15m46.317343117s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:44:46.446265212 (563h15m46.317348247s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:44:46.731558695 (563h15m46.60264174s) [upload=493833]: Done uploading segments...
Dec 29 02:44:46.731582651 (563h15m46.602665696s) [upload=493833]: Upload finished. err: <nil>
Duration: 11.270306127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration: 
34.22799921s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:48:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 34.228026631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.............................X..............................]
duration: 
3.523135913s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:48:21 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.523165892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X...]
duration: 
6.189466911s
trace: 
https://jaeger.dev.storj.io/trace/70ac13beb974b2b6
FAIL (2025-12-29 02:48:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A) context deadline exceeded
======LOGS======
ents...
Dec 29 02:44:43.889083534 (563h15m43.760166569s) [upload=493833]: Got next segment. Inline: false
Dec 29 02:44:43.889176521 (563h15m43.760259556s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:44:43.889184266 (563h15m43.760267301s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:44:44.135574794 (563h15m44.006657829s) [seg_pos={0 0}, upload=493833]: Begin upload segment...
Dec 29 02:44:44.135645478 (563h15m44.006728513s) [seg_pos={0 0}, upload=493833]: Done begin upload segment.
Dec 29 02:44:44.135697757 (563h15m44.006780802s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.135704410 (563h15m44.006787445s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136241052 (563h15m44.007324097s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136430152 (563h15m44.007513197s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.137224605 (563h15m44.00830765s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138265126 (563h15m44.009348161s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138784245 (563h15m44.00986728s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.141113890 (563h15m44.012196925s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.142571886 (563h15m44.013654921s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.143980559 (563h15m44.015063594s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.144745244 (563h15m44.015828289s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.146013931 (563h15m44.017096966s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.147269792 (563h15m44.018352827s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148364226 (563h15m44.019447261s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148841184 (563h15m44.019924249s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.150128576 (563h15m44.021211611s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:45.851835745 (563h15m45.72291878s) [upload=493833]: Next returned nil segment
Dec 29 02:44:45.851851295 (563h15m45.72293434s) [upload=493833]: Waiting for error group managing segments...
Dec 29 02:44:46.139870389 (563h15m46.010953434s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.144675569 (563h15m46.015758604s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.149426535 (563h15m46.02050957s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.150548371 (563h15m46.021631406s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155786034 (563h15m46.026869069s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155850717 (563h15m46.026933752s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.156982112 (563h15m46.028065147s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.157752429 (563h15m46.028835474s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162416520 (563h15m46.033499565s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162587144 (563h15m46.033670189s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.181878848 (563h15m46.052961883s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.369052459 (563h15m46.240135514s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445436664 (563h15m46.316519709s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445459768 (563h15m46.316542803s) [seg_pos={0 0}, upload=493833]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:44:46.445738268 (563h15m46.316821314s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection
Dec 29 02:44:46.445748708 (563h15m46.316831753s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.445882603 (563h15m46.316965638s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 29 02:44:46.445917189 (563h15m46.317000224s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446105768 (563h15m46.317188813s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection
Dec 29 02:44:46.446112530 (563h15m46.317195575s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446142588 (563h15m46.317225623s) [seg_pos={0 0}, upload=493833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:44:46.446260082 (563h15m46.317343117s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:44:46.446265212 (563h15m46.317348247s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:44:46.731558695 (563h15m46.60264174s) [upload=493833]: Done uploading segments...
Dec 29 02:44:46.731582651 (563h15m46.602665696s) [upload=493833]: Upload finished. err: <nil>
Duration: 6.189493591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [..X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X..]
duration: 
23.298939678s
trace: 
https://jaeger.dev.storj.io/trace/70ac13beb974b2b6
FAIL (2025-12-29 02:47:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A) context deadline exceeded
======LOGS======
ents...
Dec 29 02:44:43.889083534 (563h15m43.760166569s) [upload=493833]: Got next segment. Inline: false
Dec 29 02:44:43.889176521 (563h15m43.760259556s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:44:43.889184266 (563h15m43.760267301s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:44:44.135574794 (563h15m44.006657829s) [seg_pos={0 0}, upload=493833]: Begin upload segment...
Dec 29 02:44:44.135645478 (563h15m44.006728513s) [seg_pos={0 0}, upload=493833]: Done begin upload segment.
Dec 29 02:44:44.135697757 (563h15m44.006780802s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.135704410 (563h15m44.006787445s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136241052 (563h15m44.007324097s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136430152 (563h15m44.007513197s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.137224605 (563h15m44.00830765s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138265126 (563h15m44.009348161s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138784245 (563h15m44.00986728s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.141113890 (563h15m44.012196925s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.142571886 (563h15m44.013654921s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.143980559 (563h15m44.015063594s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.144745244 (563h15m44.015828289s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.146013931 (563h15m44.017096966s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.147269792 (563h15m44.018352827s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148364226 (563h15m44.019447261s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148841184 (563h15m44.019924249s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.150128576 (563h15m44.021211611s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:45.851835745 (563h15m45.72291878s) [upload=493833]: Next returned nil segment
Dec 29 02:44:45.851851295 (563h15m45.72293434s) [upload=493833]: Waiting for error group managing segments...
Dec 29 02:44:46.139870389 (563h15m46.010953434s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.144675569 (563h15m46.015758604s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.149426535 (563h15m46.02050957s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.150548371 (563h15m46.021631406s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155786034 (563h15m46.026869069s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155850717 (563h15m46.026933752s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.156982112 (563h15m46.028065147s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.157752429 (563h15m46.028835474s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162416520 (563h15m46.033499565s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162587144 (563h15m46.033670189s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.181878848 (563h15m46.052961883s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.369052459 (563h15m46.240135514s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445436664 (563h15m46.316519709s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445459768 (563h15m46.316542803s) [seg_pos={0 0}, upload=493833]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:44:46.445738268 (563h15m46.316821314s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection
Dec 29 02:44:46.445748708 (563h15m46.316831753s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.445882603 (563h15m46.316965638s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 29 02:44:46.445917189 (563h15m46.317000224s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446105768 (563h15m46.317188813s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection
Dec 29 02:44:46.446112530 (563h15m46.317195575s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446142588 (563h15m46.317225623s) [seg_pos={0 0}, upload=493833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:44:46.446260082 (563h15m46.317343117s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:44:46.446265212 (563h15m46.317348247s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:44:46.731558695 (563h15m46.60264174s) [upload=493833]: Done uploading segments...
Dec 29 02:44:46.731582651 (563h15m46.602665696s) [upload=493833]: Upload finished. err: <nil>
Duration: 23.29896697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................................X........]
duration: 
27.889326645s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:47:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 27.88934544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..............................X.............................]
duration: 
3.545499252s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:47:07 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.545529281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration: 
40.717193666s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:46:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 40.717220156s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...............................X............................]
duration: 
3.617925253s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:46:46 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.617987854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X.]
duration: 
21.468530164s
trace: 
https://jaeger.dev.storj.io/trace/70ac13beb974b2b6
FAIL (2025-12-29 02:46:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A) context deadline exceeded
======LOGS======
ents...
Dec 29 02:44:43.889083534 (563h15m43.760166569s) [upload=493833]: Got next segment. Inline: false
Dec 29 02:44:43.889176521 (563h15m43.760259556s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:44:43.889184266 (563h15m43.760267301s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:44:44.135574794 (563h15m44.006657829s) [seg_pos={0 0}, upload=493833]: Begin upload segment...
Dec 29 02:44:44.135645478 (563h15m44.006728513s) [seg_pos={0 0}, upload=493833]: Done begin upload segment.
Dec 29 02:44:44.135697757 (563h15m44.006780802s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.135704410 (563h15m44.006787445s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136241052 (563h15m44.007324097s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136430152 (563h15m44.007513197s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.137224605 (563h15m44.00830765s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138265126 (563h15m44.009348161s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138784245 (563h15m44.00986728s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.141113890 (563h15m44.012196925s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.142571886 (563h15m44.013654921s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.143980559 (563h15m44.015063594s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.144745244 (563h15m44.015828289s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.146013931 (563h15m44.017096966s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.147269792 (563h15m44.018352827s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148364226 (563h15m44.019447261s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148841184 (563h15m44.019924249s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.150128576 (563h15m44.021211611s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:45.851835745 (563h15m45.72291878s) [upload=493833]: Next returned nil segment
Dec 29 02:44:45.851851295 (563h15m45.72293434s) [upload=493833]: Waiting for error group managing segments...
Dec 29 02:44:46.139870389 (563h15m46.010953434s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.144675569 (563h15m46.015758604s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.149426535 (563h15m46.02050957s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.150548371 (563h15m46.021631406s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155786034 (563h15m46.026869069s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155850717 (563h15m46.026933752s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.156982112 (563h15m46.028065147s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.157752429 (563h15m46.028835474s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162416520 (563h15m46.033499565s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162587144 (563h15m46.033670189s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.181878848 (563h15m46.052961883s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.369052459 (563h15m46.240135514s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445436664 (563h15m46.316519709s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445459768 (563h15m46.316542803s) [seg_pos={0 0}, upload=493833]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:44:46.445738268 (563h15m46.316821314s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection
Dec 29 02:44:46.445748708 (563h15m46.316831753s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.445882603 (563h15m46.316965638s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 29 02:44:46.445917189 (563h15m46.317000224s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446105768 (563h15m46.317188813s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection
Dec 29 02:44:46.446112530 (563h15m46.317195575s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446142588 (563h15m46.317225623s) [seg_pos={0 0}, upload=493833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:44:46.446260082 (563h15m46.317343117s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:44:46.446265212 (563h15m46.317348247s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:44:46.731558695 (563h15m46.60264174s) [upload=493833]: Done uploading segments...
Dec 29 02:44:46.731582651 (563h15m46.602665696s) [upload=493833]: Upload finished. err: <nil>
Duration: 21.468559127s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [................................X...........................]
duration: 
3.597886417s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:46:06 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.597923547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration: 
38.611907408s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:45:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 38.611926035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.................................X..........................]
duration: 
3.575986398s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:45:34 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.576150471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.X]
duration: 
30.004509397s
trace: 
https://jaeger.dev.storj.io/trace/70ac13beb974b2b6
FAIL (2025-12-29 02:45:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A) context deadline exceeded
======LOGS======
ents...
Dec 29 02:44:43.889083534 (563h15m43.760166569s) [upload=493833]: Got next segment. Inline: false
Dec 29 02:44:43.889176521 (563h15m43.760259556s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:44:43.889184266 (563h15m43.760267301s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:44:44.135574794 (563h15m44.006657829s) [seg_pos={0 0}, upload=493833]: Begin upload segment...
Dec 29 02:44:44.135645478 (563h15m44.006728513s) [seg_pos={0 0}, upload=493833]: Done begin upload segment.
Dec 29 02:44:44.135697757 (563h15m44.006780802s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.135704410 (563h15m44.006787445s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136241052 (563h15m44.007324097s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.136430152 (563h15m44.007513197s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.137224605 (563h15m44.00830765s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138265126 (563h15m44.009348161s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.138784245 (563h15m44.00986728s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.141113890 (563h15m44.012196925s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.142571886 (563h15m44.013654921s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.143980559 (563h15m44.015063594s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.144745244 (563h15m44.015828289s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.146013931 (563h15m44.017096966s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.147269792 (563h15m44.018352827s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148364226 (563h15m44.019447261s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.148841184 (563h15m44.019924249s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:44.150128576 (563h15m44.021211611s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Uploading piece...
Dec 29 02:44:45.851835745 (563h15m45.72291878s) [upload=493833]: Next returned nil segment
Dec 29 02:44:45.851851295 (563h15m45.72293434s) [upload=493833]: Waiting for error group managing segments...
Dec 29 02:44:46.139870389 (563h15m46.010953434s) [noise=true , address=145.79.227.38:4979   , piece_id=MY27KXYMY6S3DP5VL74LBI7QQ2JM44MXK3A2WKEIXKFG475U5TQQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.144675569 (563h15m46.015758604s) [noise=true , address=145.79.227.6:4258    , piece_id=SX5KDBJTIJ74ZPQZ46QULZVP54FIKDDDEQVRX6I4ZAQ6EI45X5PA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.149426535 (563h15m46.02050957s) [noise=true , address=145.79.227.13:4802   , piece_id=ZZ5MCEEI5A7BWNZG3HDSRZ7JW7T5AFRO4LNTPFWSQ7KRIF3Z7FZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.150548371 (563h15m46.021631406s) [noise=true , address=145.79.227.48:4301   , piece_id=SM2QJKLZOMDBAMUR732EU2QCPMJ2RV5ZCMRNXSD7QWXF5FHQWEQA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155786034 (563h15m46.026869069s) [noise=true , address=145.79.227.7:4294    , piece_id=Z34QMER5B7ZGTQCOKOW6PQVU667RUMCI5XII2EUL65VZGQ2QVLGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.155850717 (563h15m46.026933752s) [noise=true , address=145.79.227.51:4410   , piece_id=J3OKXYEJSNNFIPMGBS4JBLCMIU5FTDPX4J76E2U7NYCNVBQYEX2A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.156982112 (563h15m46.028065147s) [noise=true , address=145.79.227.21:4473   , piece_id=TZCCZDPNMEFSAEMR3YHLFKQIBLQBRLDSF4DJC6DUGT44R72OFSZA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.157752429 (563h15m46.028835474s) [noise=true , address=145.79.227.18:4006   , piece_id=4CZ36OSGZDEUKRXXGT4WPJTDVDYYAIQPLTMBCDOXU6Y2L2UANACQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162416520 (563h15m46.033499565s) [noise=true , address=145.79.227.11:4731   , piece_id=XV5SLHGOBI7CS7UZTP42Z5W66SSNF4RGD4TENNB4IYFYWGVGY67A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.162587144 (563h15m46.033670189s) [noise=true , address=145.79.227.50:4374   , piece_id=4PFQOZFWCS4SO57IMGIYEOGFORCWGZV5E6X4K6YW2MSWNN7NXYDQ, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.181878848 (563h15m46.052961883s) [noise=true , address=145.79.227.47:4266   , piece_id=URUFMVFNYXIHZFUCOZWJDUPFCSGZHZ4HILTJQ5P6HJCTH4YCCYHA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.369052459 (563h15m46.240135514s) [noise=true , address=45.91.116.10:4082    , piece_id=GCB6MXUIUEHZNLYP7QKZ5W7YL2EIMPCINVWHXPC2KMPBKEPYJHGA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445436664 (563h15m46.316519709s) [noise=true , address=45.91.116.9:4264     , piece_id=YLFGZBFWOOY3MWWCT4XAX7UJBGHZDHZ5CLI7LJONG4UYBJHKIN5A, seg_pos={0 0}, upload=493833]: Done uploading piece. err: <nil>
Dec 29 02:44:46.445459768 (563h15m46.316542803s) [seg_pos={0 0}, upload=493833]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:44:46.445738268 (563h15m46.316821314s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44764->45.91.116.12:4228: use of closed network connection
Dec 29 02:44:46.445748708 (563h15m46.316831753s) [noise=true , address=45.91.116.12:4228    , piece_id=WLUC7ECHBB6GBR6GGF7ZQEWZB2GQKXQYJ2DZVKOWBSICDEYWRPKA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.445882603 (563h15m46.316965638s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 29 02:44:46.445917189 (563h15m46.317000224s) [noise=true , address=77.90.31.12:4590     , piece_id=WZDVIXCVKR52NY5NSUDFRSRCVSKLWXHDRKVNLHDGYGR2VQUOKX3Q, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446105768 (563h15m46.317188813s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38042->145.79.227.2:4115: use of closed network connection
Dec 29 02:44:46.446112530 (563h15m46.317195575s) [noise=true , address=145.79.227.2:4115    , piece_id=DLJQUKYMDISHSZWGBN56NPDBKKKCJMQWOZPZX4LHZL64QYFHJKLA, seg_pos={0 0}, upload=493833]: Long tail cancellation due to optimal threshold
Dec 29 02:44:46.446142588 (563h15m46.317225623s) [seg_pos={0 0}, upload=493833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:44:46.446260082 (563h15m46.317343117s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:44:46.446265212 (563h15m46.317348247s) [upload=493833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:44:46.731558695 (563h15m46.60264174s) [upload=493833]: Done uploading segments...
Dec 29 02:44:46.731582651 (563h15m46.602665696s) [upload=493833]: Upload finished. err: <nil>
Duration: 30.004535137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration: 
41.392734472s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:45:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 41.392758377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..................................X.........................]
duration: 
4.159014942s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:45:04 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.159041031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X.]
duration: 
5.888982233s
trace: 
https://jaeger.dev.storj.io/trace/34a6baafe8e93554
FAIL (2025-12-29 02:44:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: BZBGTWBIKGL3QR6HXF5ADQQVK2JFP6JV4JX52X7X5KNJEZ5YEOSA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: CFQLPDRF7KID3QLH4OYHHGGZI6ZEEXKNOENCMSUPRW4LDEK3Q3MA) context deadline exceeded
======LOGS======
Dec 29 02:43:16.322170049 (563h14m16.193253094s) [upload=493808]: Starting upload
Dec 29 02:43:16.322251255 (563h14m16.19333429s) [upload=493808]: Done starting upload
Dec 29 02:43:16.322331497 (563h14m16.193414542s) [upload=493808]: Uploading segments...
Dec 29 02:43:16.322570673 (563h14m16.193653708s) [upload=493808]: Got next segment. Inline: false
Dec 29 02:43:16.322671825 (563h14m16.19375487s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:43:16.322678107 (563h14m16.193761142s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:43:16.570706724 (563h14m16.44178976s) [seg_pos={0 0}, upload=493808]: Begin upload segment...
Dec 29 02:43:16.570748033 (563h14m16.441831078s) [seg_pos={0 0}, upload=493808]: Done begin upload segment.
Dec 29 02:43:16.570779142 (563h14m16.441862187s) [noise=true , address=145.79.227.2:4115    , piece_id=NTWXO3O6XRBZ7A2SMADDN3AT6SPDX6S2LNS3CL44RJKHYAAX34EQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.570795072 (563h14m16.441878107s) [noise=true , address=45.91.116.7:4009     , piece_id=CFQLPDRF7KID3QLH4OYHHGGZI6ZEEXKNOENCMSUPRW4LDEK3Q3MA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.571409142 (563h14m16.442492177s) [noise=true , address=145.79.227.14:4839   , piece_id=KZKG7SMEULFYC2LK5R4ALXKE3HUAGRNS6KYV2PH55ZBFXAQNOSMA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.571814483 (563h14m16.442897528s) [noise=true , address=45.91.116.13:4155    , piece_id=IBUUKDC4O4ZRKLMCBP2LDBNQ2PB3O3VGRJTDQHZLCMEIXXYKJRMQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.572184278 (563h14m16.443267313s) [noise=true , address=77.90.31.15:4697     , piece_id=OQBC6VGXLNOOM2OA4QJHT3GVIWQ4K63MGMHFTYE5XPBI2NVCUO3A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.572493476 (563h14m16.443576511s) [noise=true , address=45.91.116.9:4263     , piece_id=DQCLEGQIU4MG7WOEEGOHBDEYNLIDLN4KI6RJI55VV5NO2MYSDC7A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.572801613 (563h14m16.443884648s) [noise=true , address=145.79.227.42:4121   , piece_id=TPTEOUJYCUKDZBAHYARHAXIWB34UQZD3L666W3J4KM7V3PHVSQ6A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.573097126 (563h14m16.444180171s) [noise=true , address=145.79.227.47:4265   , piece_id=F7NANS7PYKB3VSX5VSRMYLHYET3YDQ4AFA3UDACKGXLRHRROOXZA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.573469485 (563h14m16.44455253s) [noise=true , address=145.79.227.36:4519   , piece_id=PG7WLO5NLKAXYEU2N734ES6XAOOSJYEL6IDUEDPOELAIHQQA7RQQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.574441175 (563h14m16.44552421s) [noise=true , address=145.79.227.37:4567   , piece_id=7WUXCRO3BR5745A3JAYSTHE4BYX237QBLDVEVC42V3PQBFZY7A2A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.575857231 (563h14m16.446940266s) [noise=true , address=145.79.227.43:4159   , piece_id=EJ3VKWYLNV3KQSZZIXHTFF7USTBEIWWNGHFQ2TCNX7V3PSS3J5JQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.579556445 (563h14m16.45063948s) [noise=true , address=145.79.227.45:4230   , piece_id=HQHJR5ZRVLCRRQ6FWSOHXZQOHVOGHXSXRYZPSCIWBRGDUZLRCFLA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.581394505 (563h14m16.45247754s) [noise=true , address=45.91.116.10:4082    , piece_id=BZBGTWBIKGL3QR6HXF5ADQQVK2JFP6JV4JX52X7X5KNJEZ5YEOSA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.583457022 (563h14m16.454540057s) [noise=true , address=45.91.116.4:4940     , piece_id=6Q5IVMM7LEHLPLW2RNQSQXAMCLYDECHH6RGVU6FWSDP3JAVMKJ7A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.585391937 (563h14m16.456474982s) [noise=true , address=145.79.227.51:4410   , piece_id=RKOFEVJJTQWWFRJLZDPSBGZIHW7SYOAM2DF4H6KTQVPXAQ57YY7A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.587096212 (563h14m16.458179247s) [noise=true , address=145.79.227.11:4729   , piece_id=DNBDHALMGTKGYSFFIQSEJBSPDVYFNCKF3YY3YD5PB4FRYCY2INKQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:18.234604268 (563h14m18.105687313s) [upload=493808]: Next returned nil segment
Dec 29 02:43:18.234617514 (563h14m18.105700559s) [upload=493808]: Waiting for error group managing segments...
Dec 29 02:43:18.584007404 (563h14m18.455090449s) [noise=true , address=145.79.227.42:4121   , piece_id=TPTEOUJYCUKDZBAHYARHAXIWB34UQZD3L666W3J4KM7V3PHVSQ6A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.586433654 (563h14m18.457516699s) [noise=true , address=145.79.227.11:4729   , piece_id=DNBDHALMGTKGYSFFIQSEJBSPDVYFNCKF3YY3YD5PB4FRYCY2INKQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.594208439 (563h14m18.465291474s) [noise=true , address=145.79.227.47:4265   , piece_id=F7NANS7PYKB3VSX5VSRMYLHYET3YDQ4AFA3UDACKGXLRHRROOXZA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.597200266 (563h14m18.468283311s) [noise=true , address=145.79.227.2:4115    , piece_id=NTWXO3O6XRBZ7A2SMADDN3AT6SPDX6S2LNS3CL44RJKHYAAX34EQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.608647372 (563h14m18.479730417s) [noise=true , address=145.79.227.37:4567   , piece_id=7WUXCRO3BR5745A3JAYSTHE4BYX237QBLDVEVC42V3PQBFZY7A2A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.609143617 (563h14m18.480226662s) [noise=true , address=145.79.227.36:4519   , piece_id=PG7WLO5NLKAXYEU2N734ES6XAOOSJYEL6IDUEDPOELAIHQQA7RQQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.623534469 (563h14m18.494617504s) [noise=true , address=145.79.227.51:4410   , piece_id=RKOFEVJJTQWWFRJLZDPSBGZIHW7SYOAM2DF4H6KTQVPXAQ57YY7A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.626044388 (563h14m18.497127423s) [noise=true , address=145.79.227.45:4230   , piece_id=HQHJR5ZRVLCRRQ6FWSOHXZQOHVOGHXSXRYZPSCIWBRGDUZLRCFLA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.814483018 (563h14m18.685566053s) [noise=true , address=45.91.116.7:4009     , piece_id=CFQLPDRF7KID3QLH4OYHHGGZI6ZEEXKNOENCMSUPRW4LDEK3Q3MA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.835512431 (563h14m18.706595466s) [noise=true , address=45.91.116.4:4940     , piece_id=6Q5IVMM7LEHLPLW2RNQSQXAMCLYDECHH6RGVU6FWSDP3JAVMKJ7A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.871279405 (563h14m18.74236244s) [noise=true , address=77.90.31.15:4697     , piece_id=OQBC6VGXLNOOM2OA4QJHT3GVIWQ4K63MGMHFTYE5XPBI2NVCUO3A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:19.088701865 (563h14m18.9597849s) [noise=true , address=145.79.227.14:4839   , piece_id=KZKG7SMEULFYC2LK5R4ALXKE3HUAGRNS6KYV2PH55ZBFXAQNOSMA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:19.506220141 (563h14m19.377303176s) [noise=true , address=45.91.116.10:4082    , piece_id=BZBGTWBIKGL3QR6HXF5ADQQVK2JFP6JV4JX52X7X5KNJEZ5YEOSA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:19.506239348 (563h14m19.377322393s) [seg_pos={0 0}, upload=493808]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:43:19.506497099 (563h14m19.377580144s) [noise=true , address=45.91.116.13:4155    , piece_id=IBUUKDC4O4ZRKLMCBP2LDBNQ2PB3O3VGRJTDQHZLCMEIXXYKJRMQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 29 02:43:19.506505916 (563h14m19.377588971s) [noise=true , address=45.91.116.13:4155    , piece_id=IBUUKDC4O4ZRKLMCBP2LDBNQ2PB3O3VGRJTDQHZLCMEIXXYKJRMQ, seg_pos={0 0}, upload=493808]: Long tail cancellation due to optimal threshold
Dec 29 02:43:19.506720635 (563h14m19.37780368s) [noise=true , address=145.79.227.43:4159   , piece_id=EJ3VKWYLNV3KQSZZIXHTFF7USTBEIWWNGHFQ2TCNX7V3PSS3J5JQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 29 02:43:19.506729752 (563h14m19.377812787s) [noise=true , address=145.79.227.43:4159   , piece_id=EJ3VKWYLNV3KQSZZIXHTFF7USTBEIWWNGHFQ2TCNX7V3PSS3J5JQ, seg_pos={0 0}, upload=493808]: Long tail cancellation due to optimal threshold
Dec 29 02:43:19.506931015 (563h14m19.378014061s) [noise=true , address=45.91.116.9:4263     , piece_id=DQCLEGQIU4MG7WOEEGOHBDEYNLIDLN4KI6RJI55VV5NO2MYSDC7A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 29 02:43:19.506951073 (563h14m19.378034118s) [noise=true , address=45.91.116.9:4263     , piece_id=DQCLEGQIU4MG7WOEEGOHBDEYNLIDLN4KI6RJI55VV5NO2MYSDC7A, seg_pos={0 0}, upload=493808]: Long tail cancellation due to optimal threshold
Dec 29 02:43:19.507010667 (563h14m19.378093712s) [seg_pos={0 0}, upload=493808]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:43:19.507136938 (563h14m19.378219983s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:43:19.507145003 (563h14m19.378228048s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:43:19.793379006 (563h14m19.664462051s) [upload=493808]: Done uploading segments...
Dec 29 02:43:19.793402111 (563h14m19.664485156s) [upload=493808]: Upload finished. err: <nil>
Duration: 5.889008592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration: 
20.397121685s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:44:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 20.397148987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...................................X........................]
duration: 
3.640314205s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:43:51 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.640337109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [..X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.X]
duration: 
30.003742385s
trace: 
https://jaeger.dev.storj.io/trace/34a6baafe8e93554
FAIL (2025-12-29 02:43:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: BZBGTWBIKGL3QR6HXF5ADQQVK2JFP6JV4JX52X7X5KNJEZ5YEOSA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: CFQLPDRF7KID3QLH4OYHHGGZI6ZEEXKNOENCMSUPRW4LDEK3Q3MA) context deadline exceeded
======LOGS======
Dec 29 02:43:16.322170049 (563h14m16.193253094s) [upload=493808]: Starting upload
Dec 29 02:43:16.322251255 (563h14m16.19333429s) [upload=493808]: Done starting upload
Dec 29 02:43:16.322331497 (563h14m16.193414542s) [upload=493808]: Uploading segments...
Dec 29 02:43:16.322570673 (563h14m16.193653708s) [upload=493808]: Got next segment. Inline: false
Dec 29 02:43:16.322671825 (563h14m16.19375487s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:43:16.322678107 (563h14m16.193761142s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:43:16.570706724 (563h14m16.44178976s) [seg_pos={0 0}, upload=493808]: Begin upload segment...
Dec 29 02:43:16.570748033 (563h14m16.441831078s) [seg_pos={0 0}, upload=493808]: Done begin upload segment.
Dec 29 02:43:16.570779142 (563h14m16.441862187s) [noise=true , address=145.79.227.2:4115    , piece_id=NTWXO3O6XRBZ7A2SMADDN3AT6SPDX6S2LNS3CL44RJKHYAAX34EQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.570795072 (563h14m16.441878107s) [noise=true , address=45.91.116.7:4009     , piece_id=CFQLPDRF7KID3QLH4OYHHGGZI6ZEEXKNOENCMSUPRW4LDEK3Q3MA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.571409142 (563h14m16.442492177s) [noise=true , address=145.79.227.14:4839   , piece_id=KZKG7SMEULFYC2LK5R4ALXKE3HUAGRNS6KYV2PH55ZBFXAQNOSMA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.571814483 (563h14m16.442897528s) [noise=true , address=45.91.116.13:4155    , piece_id=IBUUKDC4O4ZRKLMCBP2LDBNQ2PB3O3VGRJTDQHZLCMEIXXYKJRMQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.572184278 (563h14m16.443267313s) [noise=true , address=77.90.31.15:4697     , piece_id=OQBC6VGXLNOOM2OA4QJHT3GVIWQ4K63MGMHFTYE5XPBI2NVCUO3A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.572493476 (563h14m16.443576511s) [noise=true , address=45.91.116.9:4263     , piece_id=DQCLEGQIU4MG7WOEEGOHBDEYNLIDLN4KI6RJI55VV5NO2MYSDC7A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.572801613 (563h14m16.443884648s) [noise=true , address=145.79.227.42:4121   , piece_id=TPTEOUJYCUKDZBAHYARHAXIWB34UQZD3L666W3J4KM7V3PHVSQ6A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.573097126 (563h14m16.444180171s) [noise=true , address=145.79.227.47:4265   , piece_id=F7NANS7PYKB3VSX5VSRMYLHYET3YDQ4AFA3UDACKGXLRHRROOXZA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.573469485 (563h14m16.44455253s) [noise=true , address=145.79.227.36:4519   , piece_id=PG7WLO5NLKAXYEU2N734ES6XAOOSJYEL6IDUEDPOELAIHQQA7RQQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.574441175 (563h14m16.44552421s) [noise=true , address=145.79.227.37:4567   , piece_id=7WUXCRO3BR5745A3JAYSTHE4BYX237QBLDVEVC42V3PQBFZY7A2A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.575857231 (563h14m16.446940266s) [noise=true , address=145.79.227.43:4159   , piece_id=EJ3VKWYLNV3KQSZZIXHTFF7USTBEIWWNGHFQ2TCNX7V3PSS3J5JQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.579556445 (563h14m16.45063948s) [noise=true , address=145.79.227.45:4230   , piece_id=HQHJR5ZRVLCRRQ6FWSOHXZQOHVOGHXSXRYZPSCIWBRGDUZLRCFLA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.581394505 (563h14m16.45247754s) [noise=true , address=45.91.116.10:4082    , piece_id=BZBGTWBIKGL3QR6HXF5ADQQVK2JFP6JV4JX52X7X5KNJEZ5YEOSA, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.583457022 (563h14m16.454540057s) [noise=true , address=45.91.116.4:4940     , piece_id=6Q5IVMM7LEHLPLW2RNQSQXAMCLYDECHH6RGVU6FWSDP3JAVMKJ7A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.585391937 (563h14m16.456474982s) [noise=true , address=145.79.227.51:4410   , piece_id=RKOFEVJJTQWWFRJLZDPSBGZIHW7SYOAM2DF4H6KTQVPXAQ57YY7A, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:16.587096212 (563h14m16.458179247s) [noise=true , address=145.79.227.11:4729   , piece_id=DNBDHALMGTKGYSFFIQSEJBSPDVYFNCKF3YY3YD5PB4FRYCY2INKQ, seg_pos={0 0}, upload=493808]: Uploading piece...
Dec 29 02:43:18.234604268 (563h14m18.105687313s) [upload=493808]: Next returned nil segment
Dec 29 02:43:18.234617514 (563h14m18.105700559s) [upload=493808]: Waiting for error group managing segments...
Dec 29 02:43:18.584007404 (563h14m18.455090449s) [noise=true , address=145.79.227.42:4121   , piece_id=TPTEOUJYCUKDZBAHYARHAXIWB34UQZD3L666W3J4KM7V3PHVSQ6A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.586433654 (563h14m18.457516699s) [noise=true , address=145.79.227.11:4729   , piece_id=DNBDHALMGTKGYSFFIQSEJBSPDVYFNCKF3YY3YD5PB4FRYCY2INKQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.594208439 (563h14m18.465291474s) [noise=true , address=145.79.227.47:4265   , piece_id=F7NANS7PYKB3VSX5VSRMYLHYET3YDQ4AFA3UDACKGXLRHRROOXZA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.597200266 (563h14m18.468283311s) [noise=true , address=145.79.227.2:4115    , piece_id=NTWXO3O6XRBZ7A2SMADDN3AT6SPDX6S2LNS3CL44RJKHYAAX34EQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.608647372 (563h14m18.479730417s) [noise=true , address=145.79.227.37:4567   , piece_id=7WUXCRO3BR5745A3JAYSTHE4BYX237QBLDVEVC42V3PQBFZY7A2A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.609143617 (563h14m18.480226662s) [noise=true , address=145.79.227.36:4519   , piece_id=PG7WLO5NLKAXYEU2N734ES6XAOOSJYEL6IDUEDPOELAIHQQA7RQQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.623534469 (563h14m18.494617504s) [noise=true , address=145.79.227.51:4410   , piece_id=RKOFEVJJTQWWFRJLZDPSBGZIHW7SYOAM2DF4H6KTQVPXAQ57YY7A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.626044388 (563h14m18.497127423s) [noise=true , address=145.79.227.45:4230   , piece_id=HQHJR5ZRVLCRRQ6FWSOHXZQOHVOGHXSXRYZPSCIWBRGDUZLRCFLA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.814483018 (563h14m18.685566053s) [noise=true , address=45.91.116.7:4009     , piece_id=CFQLPDRF7KID3QLH4OYHHGGZI6ZEEXKNOENCMSUPRW4LDEK3Q3MA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.835512431 (563h14m18.706595466s) [noise=true , address=45.91.116.4:4940     , piece_id=6Q5IVMM7LEHLPLW2RNQSQXAMCLYDECHH6RGVU6FWSDP3JAVMKJ7A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:18.871279405 (563h14m18.74236244s) [noise=true , address=77.90.31.15:4697     , piece_id=OQBC6VGXLNOOM2OA4QJHT3GVIWQ4K63MGMHFTYE5XPBI2NVCUO3A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:19.088701865 (563h14m18.9597849s) [noise=true , address=145.79.227.14:4839   , piece_id=KZKG7SMEULFYC2LK5R4ALXKE3HUAGRNS6KYV2PH55ZBFXAQNOSMA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:19.506220141 (563h14m19.377303176s) [noise=true , address=45.91.116.10:4082    , piece_id=BZBGTWBIKGL3QR6HXF5ADQQVK2JFP6JV4JX52X7X5KNJEZ5YEOSA, seg_pos={0 0}, upload=493808]: Done uploading piece. err: <nil>
Dec 29 02:43:19.506239348 (563h14m19.377322393s) [seg_pos={0 0}, upload=493808]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:43:19.506497099 (563h14m19.377580144s) [noise=true , address=45.91.116.13:4155    , piece_id=IBUUKDC4O4ZRKLMCBP2LDBNQ2PB3O3VGRJTDQHZLCMEIXXYKJRMQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 29 02:43:19.506505916 (563h14m19.377588971s) [noise=true , address=45.91.116.13:4155    , piece_id=IBUUKDC4O4ZRKLMCBP2LDBNQ2PB3O3VGRJTDQHZLCMEIXXYKJRMQ, seg_pos={0 0}, upload=493808]: Long tail cancellation due to optimal threshold
Dec 29 02:43:19.506720635 (563h14m19.37780368s) [noise=true , address=145.79.227.43:4159   , piece_id=EJ3VKWYLNV3KQSZZIXHTFF7USTBEIWWNGHFQ2TCNX7V3PSS3J5JQ, seg_pos={0 0}, upload=493808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 29 02:43:19.506729752 (563h14m19.377812787s) [noise=true , address=145.79.227.43:4159   , piece_id=EJ3VKWYLNV3KQSZZIXHTFF7USTBEIWWNGHFQ2TCNX7V3PSS3J5JQ, seg_pos={0 0}, upload=493808]: Long tail cancellation due to optimal threshold
Dec 29 02:43:19.506931015 (563h14m19.378014061s) [noise=true , address=45.91.116.9:4263     , piece_id=DQCLEGQIU4MG7WOEEGOHBDEYNLIDLN4KI6RJI55VV5NO2MYSDC7A, seg_pos={0 0}, upload=493808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 29 02:43:19.506951073 (563h14m19.378034118s) [noise=true , address=45.91.116.9:4263     , piece_id=DQCLEGQIU4MG7WOEEGOHBDEYNLIDLN4KI6RJI55VV5NO2MYSDC7A, seg_pos={0 0}, upload=493808]: Long tail cancellation due to optimal threshold
Dec 29 02:43:19.507010667 (563h14m19.378093712s) [seg_pos={0 0}, upload=493808]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:43:19.507136938 (563h14m19.378219983s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:43:19.507145003 (563h14m19.378228048s) [upload=493808]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:43:19.793379006 (563h14m19.664462051s) [upload=493808]: Done uploading segments...
Dec 29 02:43:19.793402111 (563h14m19.664485156s) [upload=493808]: Upload finished. err: <nil>
Duration: 30.003817167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................................X...]
duration: 
46.443058135s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:43:40 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 46.443079174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [....................................X.......................]
duration: 
3.598803857s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:43:17 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.598833322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [...X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X.]
duration: 
24.137699626s
trace: 
https://jaeger.dev.storj.io/trace/7df291898a576125
FAIL (2025-12-29 02:42:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 02:41:39.148575251 (563h12m39.019658286s) [upload=493788]: Starting upload
Dec 29 02:41:39.148627320 (563h12m39.019710355s) [upload=493788]: Done starting upload
Dec 29 02:41:39.148697013 (563h12m39.019780058s) [upload=493788]: Uploading segments...
Dec 29 02:41:39.148800420 (563h12m39.019883455s) [upload=493788]: Got next segment. Inline: false
Dec 29 02:41:39.148829535 (563h12m39.01991257s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:41:39.148835807 (563h12m39.019918852s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:41:39.397043674 (563h12m39.26812673s) [seg_pos={0 0}, upload=493788]: Begin upload segment...
Dec 29 02:41:39.397088319 (563h12m39.268171354s) [seg_pos={0 0}, upload=493788]: Done begin upload segment.
Dec 29 02:41:39.397147662 (563h12m39.268230708s) [noise=true , address=77.90.31.10:4518     , piece_id=WIYKKKSTA6CFVPOY4I2OBUU3JKHYKH5Z5LV57SAJHG3RXAS2TPVA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.397203358 (563h12m39.268286403s) [noise=true , address=45.91.116.6:4976     , piece_id=FAUNFH6ADE27HELMOPMYV5OMZ7PVKP5OE3HEKKUVA4UDLPDNZDPA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.397823469 (563h12m39.268906504s) [noise=true , address=45.91.116.7:4010     , piece_id=WJXJ3MBYGIQLOFRA3WTEDJ5PHQHDLUJYNCINFYSW7K5DD44OS66Q, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.398214393 (563h12m39.269297428s) [noise=true , address=45.91.116.10:4083    , piece_id=5XES33NNSIU3252VVDBWZR7TXXRPZSNBWDT4PFTJBHN4XTRI4OKA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.398552056 (563h12m39.269635091s) [noise=true , address=145.79.227.12:4767   , piece_id=MN6WTCZG2FRL5GXWNZICY6H2DYRECRJZV4TRIWDNEHQX5MTMNHOQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.398835957 (563h12m39.269919002s) [noise=true , address=145.79.227.26:4654   , piece_id=WJ2R2SVLF6J6AUB4T6VSWD4RWOBN2SLVI33ZDLOJXNBN6KFJFV4A, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.399273210 (563h12m39.270356245s) [noise=true , address=45.91.116.5:4300     , piece_id=AR2V5BNMP2C6FLYVW63T4YBEBQDBYRUAYMOZTQXU72UHHTGV2IIA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.400481149 (563h12m39.271564184s) [noise=true , address=145.79.227.50:4374   , piece_id=5654L3G4MU2F5UFN7JNAPRD376ZM4AHHWI3FPF7GZRAWR5ARD6VQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.401368019 (563h12m39.272451064s) [noise=true , address=145.79.227.52:4447   , piece_id=EYPA6YLWSQEUFW2LEWC4GYD73UA4FEDUUERHJOPOWBDNM5HXDFTA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.401964254 (563h12m39.273047299s) [noise=true , address=45.91.116.8:4045     , piece_id=DYOEFN7MGZLWFD4OYE3Q2TQ3XPPFUKEFDMM6NGPZTPORLXKCO6BA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.403511580 (563h12m39.274594615s) [noise=true , address=145.79.227.51:4410   , piece_id=HQDRIHBDBGXRY3NIJUCOSU2W73VD654UZMH34AYFG24J2DOPP6AQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.405749711 (563h12m39.276832756s) [noise=true , address=145.79.227.48:4301   , piece_id=JZ5JYUC27CJCSFW4ITUJR4BZZA7I2BFT6HRWA2E3XJVHIFLQU53A, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.408330085 (563h12m39.279413121s) [noise=true , address=77.90.31.11:4556     , piece_id=F2SME5FTKAQH6O44VP7EFCN4TMZQTYQGA67LE6NWRMYIP27LNG3Q, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.411027411 (563h12m39.282110446s) [noise=true , address=45.91.116.4:4940     , piece_id=NQQJKRF74M72MKO24KYSFADBHRJ6LA2XLKHJPWILHONOMDTJ7GRQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.412245210 (563h12m39.283328255s) [noise=true , address=145.79.227.14:4840   , piece_id=7SAQM2ENZG3OYDQ7MGCB7HYZPLY2AOHBS3ZKEQGBIZHEW3X25M7Q, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.413718836 (563h12m39.284801881s) [noise=true , address=145.79.227.17:4972   , piece_id=CF3EKEWHUGK63QLKF2YDTGMTPVT4P3VDGO4DWNWEMLHCND3JRYMA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:41.126130778 (563h12m40.997213833s) [upload=493788]: Next returned nil segment
Dec 29 02:41:41.126147781 (563h12m40.997230816s) [upload=493788]: Waiting for error group managing segments...
Dec 29 02:41:41.404897365 (563h12m41.2759804s) [noise=true , address=145.79.227.26:4654   , piece_id=WJ2R2SVLF6J6AUB4T6VSWD4RWOBN2SLVI33ZDLOJXNBN6KFJFV4A, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.411919655 (563h12m41.283002701s) [noise=true , address=145.79.227.17:4972   , piece_id=CF3EKEWHUGK63QLKF2YDTGMTPVT4P3VDGO4DWNWEMLHCND3JRYMA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.413764789 (563h12m41.284847824s) [noise=true , address=145.79.227.51:4410   , piece_id=HQDRIHBDBGXRY3NIJUCOSU2W73VD654UZMH34AYFG24J2DOPP6AQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.414782366 (563h12m41.285865411s) [noise=true , address=145.79.227.12:4767   , piece_id=MN6WTCZG2FRL5GXWNZICY6H2DYRECRJZV4TRIWDNEHQX5MTMNHOQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.416238169 (563h12m41.287321204s) [noise=true , address=145.79.227.48:4301   , piece_id=JZ5JYUC27CJCSFW4ITUJR4BZZA7I2BFT6HRWA2E3XJVHIFLQU53A, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.418850303 (563h12m41.289933348s) [noise=true , address=145.79.227.50:4374   , piece_id=5654L3G4MU2F5UFN7JNAPRD376ZM4AHHWI3FPF7GZRAWR5ARD6VQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.436350074 (563h12m41.307433119s) [noise=true , address=145.79.227.14:4840   , piece_id=7SAQM2ENZG3OYDQ7MGCB7HYZPLY2AOHBS3ZKEQGBIZHEW3X25M7Q, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.459795567 (563h12m41.330878602s) [noise=true , address=145.79.227.52:4447   , piece_id=EYPA6YLWSQEUFW2LEWC4GYD73UA4FEDUUERHJOPOWBDNM5HXDFTA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.617932598 (563h12m41.489015643s) [noise=true , address=45.91.116.5:4300     , piece_id=AR2V5BNMP2C6FLYVW63T4YBEBQDBYRUAYMOZTQXU72UHHTGV2IIA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.658332374 (563h12m41.529415419s) [noise=true , address=45.91.116.10:4083    , piece_id=5XES33NNSIU3252VVDBWZR7TXXRPZSNBWDT4PFTJBHN4XTRI4OKA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.671621408 (563h12m41.542704443s) [noise=true , address=45.91.116.6:4976     , piece_id=FAUNFH6ADE27HELMOPMYV5OMZ7PVKP5OE3HEKKUVA4UDLPDNZDPA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.673348327 (563h12m41.544431372s) [noise=true , address=45.91.116.8:4045     , piece_id=DYOEFN7MGZLWFD4OYE3Q2TQ3XPPFUKEFDMM6NGPZTPORLXKCO6BA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.681307893 (563h12m41.552390938s) [noise=true , address=77.90.31.11:4556     , piece_id=F2SME5FTKAQH6O44VP7EFCN4TMZQTYQGA67LE6NWRMYIP27LNG3Q, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.681327290 (563h12m41.552410335s) [seg_pos={0 0}, upload=493788]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:41:41.681507914 (563h12m41.552590949s) [noise=true , address=45.91.116.4:4940     , piece_id=NQQJKRF74M72MKO24KYSFADBHRJ6LA2XLKHJPWILHONOMDTJ7GRQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 02:41:41.681514456 (563h12m41.552597491s) [noise=true , address=45.91.116.4:4940     , piece_id=NQQJKRF74M72MKO24KYSFADBHRJ6LA2XLKHJPWILHONOMDTJ7GRQ, seg_pos={0 0}, upload=493788]: Long tail cancellation due to optimal threshold
Dec 29 02:41:41.681663640 (563h12m41.552746675s) [noise=true , address=77.90.31.10:4518     , piece_id=WIYKKKSTA6CFVPOY4I2OBUU3JKHYKH5Z5LV57SAJHG3RXAS2TPVA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 29 02:41:41.681688117 (563h12m41.552771162s) [noise=true , address=77.90.31.10:4518     , piece_id=WIYKKKSTA6CFVPOY4I2OBUU3JKHYKH5Z5LV57SAJHG3RXAS2TPVA, seg_pos={0 0}, upload=493788]: Long tail cancellation due to optimal threshold
Dec 29 02:41:41.681739815 (563h12m41.55282285s) [noise=true , address=45.91.116.7:4010     , piece_id=WJXJ3MBYGIQLOFRA3WTEDJ5PHQHDLUJYNCINFYSW7K5DD44OS66Q, seg_pos={0 0}, upload=493788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 29 02:41:41.681746929 (563h12m41.552829964s) [noise=true , address=45.91.116.7:4010     , piece_id=WJXJ3MBYGIQLOFRA3WTEDJ5PHQHDLUJYNCINFYSW7K5DD44OS66Q, seg_pos={0 0}, upload=493788]: Long tail cancellation due to optimal threshold
Dec 29 02:41:41.681786815 (563h12m41.55286985s) [seg_pos={0 0}, upload=493788]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:41:41.681925499 (563h12m41.553008534s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:41:41.681932151 (563h12m41.553015196s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:41:41.985502246 (563h12m41.856585301s) [upload=493788]: Done uploading segments...
Dec 29 02:41:41.985530299 (563h12m41.856613344s) [upload=493788]: Upload finished. err: <nil>
Duration: 24.137727889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
35.431155417s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:42:40 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 35.431174474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.....................................X......................]
duration: 
3.523607138s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:42:39 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.523667023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [......................................X.....................]
duration: 
3.607527042s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:42:11 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.607556699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [....X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..X]
duration: 
30.003846203s
trace: 
https://jaeger.dev.storj.io/trace/7df291898a576125
FAIL (2025-12-29 02:42:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 02:41:39.148575251 (563h12m39.019658286s) [upload=493788]: Starting upload
Dec 29 02:41:39.148627320 (563h12m39.019710355s) [upload=493788]: Done starting upload
Dec 29 02:41:39.148697013 (563h12m39.019780058s) [upload=493788]: Uploading segments...
Dec 29 02:41:39.148800420 (563h12m39.019883455s) [upload=493788]: Got next segment. Inline: false
Dec 29 02:41:39.148829535 (563h12m39.01991257s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:41:39.148835807 (563h12m39.019918852s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:41:39.397043674 (563h12m39.26812673s) [seg_pos={0 0}, upload=493788]: Begin upload segment...
Dec 29 02:41:39.397088319 (563h12m39.268171354s) [seg_pos={0 0}, upload=493788]: Done begin upload segment.
Dec 29 02:41:39.397147662 (563h12m39.268230708s) [noise=true , address=77.90.31.10:4518     , piece_id=WIYKKKSTA6CFVPOY4I2OBUU3JKHYKH5Z5LV57SAJHG3RXAS2TPVA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.397203358 (563h12m39.268286403s) [noise=true , address=45.91.116.6:4976     , piece_id=FAUNFH6ADE27HELMOPMYV5OMZ7PVKP5OE3HEKKUVA4UDLPDNZDPA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.397823469 (563h12m39.268906504s) [noise=true , address=45.91.116.7:4010     , piece_id=WJXJ3MBYGIQLOFRA3WTEDJ5PHQHDLUJYNCINFYSW7K5DD44OS66Q, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.398214393 (563h12m39.269297428s) [noise=true , address=45.91.116.10:4083    , piece_id=5XES33NNSIU3252VVDBWZR7TXXRPZSNBWDT4PFTJBHN4XTRI4OKA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.398552056 (563h12m39.269635091s) [noise=true , address=145.79.227.12:4767   , piece_id=MN6WTCZG2FRL5GXWNZICY6H2DYRECRJZV4TRIWDNEHQX5MTMNHOQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.398835957 (563h12m39.269919002s) [noise=true , address=145.79.227.26:4654   , piece_id=WJ2R2SVLF6J6AUB4T6VSWD4RWOBN2SLVI33ZDLOJXNBN6KFJFV4A, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.399273210 (563h12m39.270356245s) [noise=true , address=45.91.116.5:4300     , piece_id=AR2V5BNMP2C6FLYVW63T4YBEBQDBYRUAYMOZTQXU72UHHTGV2IIA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.400481149 (563h12m39.271564184s) [noise=true , address=145.79.227.50:4374   , piece_id=5654L3G4MU2F5UFN7JNAPRD376ZM4AHHWI3FPF7GZRAWR5ARD6VQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.401368019 (563h12m39.272451064s) [noise=true , address=145.79.227.52:4447   , piece_id=EYPA6YLWSQEUFW2LEWC4GYD73UA4FEDUUERHJOPOWBDNM5HXDFTA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.401964254 (563h12m39.273047299s) [noise=true , address=45.91.116.8:4045     , piece_id=DYOEFN7MGZLWFD4OYE3Q2TQ3XPPFUKEFDMM6NGPZTPORLXKCO6BA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.403511580 (563h12m39.274594615s) [noise=true , address=145.79.227.51:4410   , piece_id=HQDRIHBDBGXRY3NIJUCOSU2W73VD654UZMH34AYFG24J2DOPP6AQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.405749711 (563h12m39.276832756s) [noise=true , address=145.79.227.48:4301   , piece_id=JZ5JYUC27CJCSFW4ITUJR4BZZA7I2BFT6HRWA2E3XJVHIFLQU53A, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.408330085 (563h12m39.279413121s) [noise=true , address=77.90.31.11:4556     , piece_id=F2SME5FTKAQH6O44VP7EFCN4TMZQTYQGA67LE6NWRMYIP27LNG3Q, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.411027411 (563h12m39.282110446s) [noise=true , address=45.91.116.4:4940     , piece_id=NQQJKRF74M72MKO24KYSFADBHRJ6LA2XLKHJPWILHONOMDTJ7GRQ, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.412245210 (563h12m39.283328255s) [noise=true , address=145.79.227.14:4840   , piece_id=7SAQM2ENZG3OYDQ7MGCB7HYZPLY2AOHBS3ZKEQGBIZHEW3X25M7Q, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:39.413718836 (563h12m39.284801881s) [noise=true , address=145.79.227.17:4972   , piece_id=CF3EKEWHUGK63QLKF2YDTGMTPVT4P3VDGO4DWNWEMLHCND3JRYMA, seg_pos={0 0}, upload=493788]: Uploading piece...
Dec 29 02:41:41.126130778 (563h12m40.997213833s) [upload=493788]: Next returned nil segment
Dec 29 02:41:41.126147781 (563h12m40.997230816s) [upload=493788]: Waiting for error group managing segments...
Dec 29 02:41:41.404897365 (563h12m41.2759804s) [noise=true , address=145.79.227.26:4654   , piece_id=WJ2R2SVLF6J6AUB4T6VSWD4RWOBN2SLVI33ZDLOJXNBN6KFJFV4A, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.411919655 (563h12m41.283002701s) [noise=true , address=145.79.227.17:4972   , piece_id=CF3EKEWHUGK63QLKF2YDTGMTPVT4P3VDGO4DWNWEMLHCND3JRYMA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.413764789 (563h12m41.284847824s) [noise=true , address=145.79.227.51:4410   , piece_id=HQDRIHBDBGXRY3NIJUCOSU2W73VD654UZMH34AYFG24J2DOPP6AQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.414782366 (563h12m41.285865411s) [noise=true , address=145.79.227.12:4767   , piece_id=MN6WTCZG2FRL5GXWNZICY6H2DYRECRJZV4TRIWDNEHQX5MTMNHOQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.416238169 (563h12m41.287321204s) [noise=true , address=145.79.227.48:4301   , piece_id=JZ5JYUC27CJCSFW4ITUJR4BZZA7I2BFT6HRWA2E3XJVHIFLQU53A, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.418850303 (563h12m41.289933348s) [noise=true , address=145.79.227.50:4374   , piece_id=5654L3G4MU2F5UFN7JNAPRD376ZM4AHHWI3FPF7GZRAWR5ARD6VQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.436350074 (563h12m41.307433119s) [noise=true , address=145.79.227.14:4840   , piece_id=7SAQM2ENZG3OYDQ7MGCB7HYZPLY2AOHBS3ZKEQGBIZHEW3X25M7Q, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.459795567 (563h12m41.330878602s) [noise=true , address=145.79.227.52:4447   , piece_id=EYPA6YLWSQEUFW2LEWC4GYD73UA4FEDUUERHJOPOWBDNM5HXDFTA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.617932598 (563h12m41.489015643s) [noise=true , address=45.91.116.5:4300     , piece_id=AR2V5BNMP2C6FLYVW63T4YBEBQDBYRUAYMOZTQXU72UHHTGV2IIA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.658332374 (563h12m41.529415419s) [noise=true , address=45.91.116.10:4083    , piece_id=5XES33NNSIU3252VVDBWZR7TXXRPZSNBWDT4PFTJBHN4XTRI4OKA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.671621408 (563h12m41.542704443s) [noise=true , address=45.91.116.6:4976     , piece_id=FAUNFH6ADE27HELMOPMYV5OMZ7PVKP5OE3HEKKUVA4UDLPDNZDPA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.673348327 (563h12m41.544431372s) [noise=true , address=45.91.116.8:4045     , piece_id=DYOEFN7MGZLWFD4OYE3Q2TQ3XPPFUKEFDMM6NGPZTPORLXKCO6BA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.681307893 (563h12m41.552390938s) [noise=true , address=77.90.31.11:4556     , piece_id=F2SME5FTKAQH6O44VP7EFCN4TMZQTYQGA67LE6NWRMYIP27LNG3Q, seg_pos={0 0}, upload=493788]: Done uploading piece. err: <nil>
Dec 29 02:41:41.681327290 (563h12m41.552410335s) [seg_pos={0 0}, upload=493788]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:41:41.681507914 (563h12m41.552590949s) [noise=true , address=45.91.116.4:4940     , piece_id=NQQJKRF74M72MKO24KYSFADBHRJ6LA2XLKHJPWILHONOMDTJ7GRQ, seg_pos={0 0}, upload=493788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 02:41:41.681514456 (563h12m41.552597491s) [noise=true , address=45.91.116.4:4940     , piece_id=NQQJKRF74M72MKO24KYSFADBHRJ6LA2XLKHJPWILHONOMDTJ7GRQ, seg_pos={0 0}, upload=493788]: Long tail cancellation due to optimal threshold
Dec 29 02:41:41.681663640 (563h12m41.552746675s) [noise=true , address=77.90.31.10:4518     , piece_id=WIYKKKSTA6CFVPOY4I2OBUU3JKHYKH5Z5LV57SAJHG3RXAS2TPVA, seg_pos={0 0}, upload=493788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 29 02:41:41.681688117 (563h12m41.552771162s) [noise=true , address=77.90.31.10:4518     , piece_id=WIYKKKSTA6CFVPOY4I2OBUU3JKHYKH5Z5LV57SAJHG3RXAS2TPVA, seg_pos={0 0}, upload=493788]: Long tail cancellation due to optimal threshold
Dec 29 02:41:41.681739815 (563h12m41.55282285s) [noise=true , address=45.91.116.7:4010     , piece_id=WJXJ3MBYGIQLOFRA3WTEDJ5PHQHDLUJYNCINFYSW7K5DD44OS66Q, seg_pos={0 0}, upload=493788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 29 02:41:41.681746929 (563h12m41.552829964s) [noise=true , address=45.91.116.7:4010     , piece_id=WJXJ3MBYGIQLOFRA3WTEDJ5PHQHDLUJYNCINFYSW7K5DD44OS66Q, seg_pos={0 0}, upload=493788]: Long tail cancellation due to optimal threshold
Dec 29 02:41:41.681786815 (563h12m41.55286985s) [seg_pos={0 0}, upload=493788]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:41:41.681925499 (563h12m41.553008534s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:41:41.681932151 (563h12m41.553015196s) [upload=493788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:41:41.985502246 (563h12m41.856585301s) [upload=493788]: Done uploading segments...
Dec 29 02:41:41.985530299 (563h12m41.856613344s) [upload=493788]: Upload finished. err: <nil>
Duration: 30.003870089s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
19.348901678s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:41:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 19.34891826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.......................................X....................]
duration: 
3.526080723s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:41:37 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.526108657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
1m0.00494456s
trace: 
https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d
FAIL (2025-12-29 02:41:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece...
Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment
Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments...
Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620   , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248    , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218    , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553   , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011      , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990     , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065     , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569     , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071   , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885     , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161   , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504   , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217   , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936   , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil>
Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces.
Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723     , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324     , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF
Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775   , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection
Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349   , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold
Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments...
Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil>
Duration: 1m0.004969138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.....X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX..]
duration: 
5.935684828s
trace: 
https://jaeger.dev.storj.io/trace/51bf53c7068cf40a
FAIL (2025-12-29 02:41:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ) context deadline exceeded
======LOGS======
gments...
Dec 29 02:39:53.777275456 (563h10m53.648358491s) [upload=493762]: Got next segment. Inline: false
Dec 29 02:39:53.777327304 (563h10m53.648410349s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:39:53.777410803 (563h10m53.648493849s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:39:54.023748009 (563h10m53.894831064s) [seg_pos={0 0}, upload=493762]: Begin upload segment...
Dec 29 02:39:54.023801421 (563h10m53.894884466s) [seg_pos={0 0}, upload=493762]: Done begin upload segment.
Dec 29 02:39:54.023834684 (563h10m53.894917719s) [noise=true , address=77.90.31.21:4880     , piece_id=WOPLT7WKIWNSAUFVKF7UWT6ALRBZVAXWILGAS7GA564GB4ET23DA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.023842238 (563h10m53.894925283s) [noise=true , address=45.91.116.8:4045     , piece_id=CJXHSD7QWZ64IXKM32VJSNS6JJPU4FDPPBCMT4PKU5E5LT7C3OBQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024388709 (563h10m53.895471754s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024673161 (563h10m53.895756206s) [noise=true , address=45.91.116.10:4082    , piece_id=H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024862290 (563h10m53.895945335s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025154277 (563h10m53.896237312s) [noise=true , address=145.79.227.32:4834   , piece_id=FJ6XR4NWFRSM6P5XVYDQLUIEMUSTTMTO7ELXTUCLM4RJDXNNVCDQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025293843 (563h10m53.896376878s) [noise=true , address=45.91.116.9:4264     , piece_id=BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025492059 (563h10m53.896575105s) [noise=true , address=145.79.227.15:4873   , piece_id=VZEHORCCR7LRA66YDAINKLO64V7SVEX2CBLLXELD3TNI6M6S3XBA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025707550 (563h10m53.896790605s) [noise=true , address=145.79.227.19:4078   , piece_id=2HBNHILFJYFTUHDH4G72TG4AIXX2P2HUVLC62HEJ75MY7JMDL6YQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025861543 (563h10m53.896944588s) [noise=true , address=145.79.227.47:4265   , piece_id=5PW5RL3AXSJBJCHDWRHT4JVD4CWUMYINDERV6OSQJKFUIGG256YA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026158829 (563h10m53.897241874s) [noise=true , address=145.79.227.51:4409   , piece_id=UX2EYGB2NF45FJREOB2UQZAAVEGGEKGZS5QNVT2R5J47GQMT2HVA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026390500 (563h10m53.897473545s) [noise=true , address=145.79.227.14:4840   , piece_id=COQ7TGCCQXNUZRA63RUAP4M2HTCUIG2C4XICF5BKWOQ7BPQEKGHQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026573509 (563h10m53.897656544s) [noise=true , address=145.79.227.16:4910   , piece_id=7TYQ4NWXORTLOGNOTPKNC4F2HKGTYTSVPUOMBKW7R5ZXORUDHKEQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026859102 (563h10m53.897942147s) [noise=true , address=145.79.227.6:4258    , piece_id=ZUE3RX265SY2DPC2NFZTO4T6HQ3EJQCVLQQNRRJVISTULGUX5PRQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.027126732 (563h10m53.898209777s) [noise=true , address=145.79.227.39:4013   , piece_id=TRJBFHI6JYRUJWKRHHYAEG5TORM3FT6S53K4NKSI3YU5WXRXAUVQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.027546471 (563h10m53.898629516s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:55.749109939 (563h10m55.620192984s) [upload=493762]: Next returned nil segment
Dec 29 02:39:55.749125449 (563h10m55.620208484s) [upload=493762]: Waiting for error group managing segments...
Dec 29 02:39:56.038419480 (563h10m55.909502515s) [noise=true , address=145.79.227.6:4258    , piece_id=ZUE3RX265SY2DPC2NFZTO4T6HQ3EJQCVLQQNRRJVISTULGUX5PRQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.039202140 (563h10m55.910285185s) [noise=true , address=145.79.227.14:4840   , piece_id=COQ7TGCCQXNUZRA63RUAP4M2HTCUIG2C4XICF5BKWOQ7BPQEKGHQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.046728070 (563h10m55.917811105s) [noise=true , address=145.79.227.51:4409   , piece_id=UX2EYGB2NF45FJREOB2UQZAAVEGGEKGZS5QNVT2R5J47GQMT2HVA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.056835877 (563h10m55.927918922s) [noise=true , address=145.79.227.15:4873   , piece_id=VZEHORCCR7LRA66YDAINKLO64V7SVEX2CBLLXELD3TNI6M6S3XBA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.089295214 (563h10m55.960378249s) [noise=true , address=145.79.227.32:4834   , piece_id=FJ6XR4NWFRSM6P5XVYDQLUIEMUSTTMTO7ELXTUCLM4RJDXNNVCDQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.089598632 (563h10m55.960681667s) [noise=true , address=145.79.227.16:4910   , piece_id=7TYQ4NWXORTLOGNOTPKNC4F2HKGTYTSVPUOMBKW7R5ZXORUDHKEQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.113200221 (563h10m55.984283256s) [noise=true , address=145.79.227.19:4078   , piece_id=2HBNHILFJYFTUHDH4G72TG4AIXX2P2HUVLC62HEJ75MY7JMDL6YQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.125066026 (563h10m55.996149071s) [noise=true , address=145.79.227.39:4013   , piece_id=TRJBFHI6JYRUJWKRHHYAEG5TORM3FT6S53K4NKSI3YU5WXRXAUVQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.144873181 (563h10m56.015956216s) [noise=true , address=145.79.227.47:4265   , piece_id=5PW5RL3AXSJBJCHDWRHT4JVD4CWUMYINDERV6OSQJKFUIGG256YA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.280035867 (563h10m56.151118902s) [noise=true , address=45.91.116.9:4264     , piece_id=BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.339585787 (563h10m56.210668822s) [noise=true , address=77.90.31.21:4880     , piece_id=WOPLT7WKIWNSAUFVKF7UWT6ALRBZVAXWILGAS7GA564GB4ET23DA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.411543494 (563h10m56.282626529s) [noise=true , address=45.91.116.10:4082    , piece_id=H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.419290625 (563h10m56.29037366s) [noise=true , address=45.91.116.8:4045     , piece_id=CJXHSD7QWZ64IXKM32VJSNS6JJPU4FDPPBCMT4PKU5E5LT7C3OBQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.419343947 (563h10m56.290426982s) [seg_pos={0 0}, upload=493762]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:39:56.419513470 (563h10m56.290596515s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 02:39:56.419522888 (563h10m56.290605923s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419605014 (563h10m56.290688049s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection
Dec 29 02:39:56.419619622 (563h10m56.290702657s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419793293 (563h10m56.290876328s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection
Dec 29 02:39:56.419800396 (563h10m56.290883431s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419815314 (563h10m56.290898349s) [seg_pos={0 0}, upload=493762]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:39:56.419911388 (563h10m56.290994453s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:39:56.419919242 (563h10m56.291002287s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:39:56.724029667 (563h10m56.595112702s) [upload=493762]: Done uploading segments...
Dec 29 02:39:56.724049535 (563h10m56.59513257s) [upload=493762]: Upload finished. err: <nil>
Duration: 5.93571156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [......X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX.]
duration: 
6.203792783s
trace: 
https://jaeger.dev.storj.io/trace/51bf53c7068cf40a
FAIL (2025-12-29 02:40:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ) context deadline exceeded
======LOGS======
gments...
Dec 29 02:39:53.777275456 (563h10m53.648358491s) [upload=493762]: Got next segment. Inline: false
Dec 29 02:39:53.777327304 (563h10m53.648410349s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:39:53.777410803 (563h10m53.648493849s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:39:54.023748009 (563h10m53.894831064s) [seg_pos={0 0}, upload=493762]: Begin upload segment...
Dec 29 02:39:54.023801421 (563h10m53.894884466s) [seg_pos={0 0}, upload=493762]: Done begin upload segment.
Dec 29 02:39:54.023834684 (563h10m53.894917719s) [noise=true , address=77.90.31.21:4880     , piece_id=WOPLT7WKIWNSAUFVKF7UWT6ALRBZVAXWILGAS7GA564GB4ET23DA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.023842238 (563h10m53.894925283s) [noise=true , address=45.91.116.8:4045     , piece_id=CJXHSD7QWZ64IXKM32VJSNS6JJPU4FDPPBCMT4PKU5E5LT7C3OBQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024388709 (563h10m53.895471754s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024673161 (563h10m53.895756206s) [noise=true , address=45.91.116.10:4082    , piece_id=H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024862290 (563h10m53.895945335s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025154277 (563h10m53.896237312s) [noise=true , address=145.79.227.32:4834   , piece_id=FJ6XR4NWFRSM6P5XVYDQLUIEMUSTTMTO7ELXTUCLM4RJDXNNVCDQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025293843 (563h10m53.896376878s) [noise=true , address=45.91.116.9:4264     , piece_id=BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025492059 (563h10m53.896575105s) [noise=true , address=145.79.227.15:4873   , piece_id=VZEHORCCR7LRA66YDAINKLO64V7SVEX2CBLLXELD3TNI6M6S3XBA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025707550 (563h10m53.896790605s) [noise=true , address=145.79.227.19:4078   , piece_id=2HBNHILFJYFTUHDH4G72TG4AIXX2P2HUVLC62HEJ75MY7JMDL6YQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025861543 (563h10m53.896944588s) [noise=true , address=145.79.227.47:4265   , piece_id=5PW5RL3AXSJBJCHDWRHT4JVD4CWUMYINDERV6OSQJKFUIGG256YA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026158829 (563h10m53.897241874s) [noise=true , address=145.79.227.51:4409   , piece_id=UX2EYGB2NF45FJREOB2UQZAAVEGGEKGZS5QNVT2R5J47GQMT2HVA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026390500 (563h10m53.897473545s) [noise=true , address=145.79.227.14:4840   , piece_id=COQ7TGCCQXNUZRA63RUAP4M2HTCUIG2C4XICF5BKWOQ7BPQEKGHQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026573509 (563h10m53.897656544s) [noise=true , address=145.79.227.16:4910   , piece_id=7TYQ4NWXORTLOGNOTPKNC4F2HKGTYTSVPUOMBKW7R5ZXORUDHKEQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026859102 (563h10m53.897942147s) [noise=true , address=145.79.227.6:4258    , piece_id=ZUE3RX265SY2DPC2NFZTO4T6HQ3EJQCVLQQNRRJVISTULGUX5PRQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.027126732 (563h10m53.898209777s) [noise=true , address=145.79.227.39:4013   , piece_id=TRJBFHI6JYRUJWKRHHYAEG5TORM3FT6S53K4NKSI3YU5WXRXAUVQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.027546471 (563h10m53.898629516s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:55.749109939 (563h10m55.620192984s) [upload=493762]: Next returned nil segment
Dec 29 02:39:55.749125449 (563h10m55.620208484s) [upload=493762]: Waiting for error group managing segments...
Dec 29 02:39:56.038419480 (563h10m55.909502515s) [noise=true , address=145.79.227.6:4258    , piece_id=ZUE3RX265SY2DPC2NFZTO4T6HQ3EJQCVLQQNRRJVISTULGUX5PRQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.039202140 (563h10m55.910285185s) [noise=true , address=145.79.227.14:4840   , piece_id=COQ7TGCCQXNUZRA63RUAP4M2HTCUIG2C4XICF5BKWOQ7BPQEKGHQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.046728070 (563h10m55.917811105s) [noise=true , address=145.79.227.51:4409   , piece_id=UX2EYGB2NF45FJREOB2UQZAAVEGGEKGZS5QNVT2R5J47GQMT2HVA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.056835877 (563h10m55.927918922s) [noise=true , address=145.79.227.15:4873   , piece_id=VZEHORCCR7LRA66YDAINKLO64V7SVEX2CBLLXELD3TNI6M6S3XBA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.089295214 (563h10m55.960378249s) [noise=true , address=145.79.227.32:4834   , piece_id=FJ6XR4NWFRSM6P5XVYDQLUIEMUSTTMTO7ELXTUCLM4RJDXNNVCDQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.089598632 (563h10m55.960681667s) [noise=true , address=145.79.227.16:4910   , piece_id=7TYQ4NWXORTLOGNOTPKNC4F2HKGTYTSVPUOMBKW7R5ZXORUDHKEQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.113200221 (563h10m55.984283256s) [noise=true , address=145.79.227.19:4078   , piece_id=2HBNHILFJYFTUHDH4G72TG4AIXX2P2HUVLC62HEJ75MY7JMDL6YQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.125066026 (563h10m55.996149071s) [noise=true , address=145.79.227.39:4013   , piece_id=TRJBFHI6JYRUJWKRHHYAEG5TORM3FT6S53K4NKSI3YU5WXRXAUVQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.144873181 (563h10m56.015956216s) [noise=true , address=145.79.227.47:4265   , piece_id=5PW5RL3AXSJBJCHDWRHT4JVD4CWUMYINDERV6OSQJKFUIGG256YA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.280035867 (563h10m56.151118902s) [noise=true , address=45.91.116.9:4264     , piece_id=BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.339585787 (563h10m56.210668822s) [noise=true , address=77.90.31.21:4880     , piece_id=WOPLT7WKIWNSAUFVKF7UWT6ALRBZVAXWILGAS7GA564GB4ET23DA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.411543494 (563h10m56.282626529s) [noise=true , address=45.91.116.10:4082    , piece_id=H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.419290625 (563h10m56.29037366s) [noise=true , address=45.91.116.8:4045     , piece_id=CJXHSD7QWZ64IXKM32VJSNS6JJPU4FDPPBCMT4PKU5E5LT7C3OBQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.419343947 (563h10m56.290426982s) [seg_pos={0 0}, upload=493762]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:39:56.419513470 (563h10m56.290596515s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 02:39:56.419522888 (563h10m56.290605923s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419605014 (563h10m56.290688049s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection
Dec 29 02:39:56.419619622 (563h10m56.290702657s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419793293 (563h10m56.290876328s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection
Dec 29 02:39:56.419800396 (563h10m56.290883431s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419815314 (563h10m56.290898349s) [seg_pos={0 0}, upload=493762]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:39:56.419911388 (563h10m56.290994453s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:39:56.419919242 (563h10m56.291002287s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:39:56.724029667 (563h10m56.595112702s) [upload=493762]: Done uploading segments...
Dec 29 02:39:56.724049535 (563h10m56.59513257s) [upload=493762]: Upload finished. err: <nil>
Duration: 6.203820716s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [........................................X...................]
duration: 
3.519805503s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:40:30 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.519838526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [X......X...X....X.....X.........X.X.....XX...XX.XXX..X.X..XX]
duration: 
30.003926561s
trace: 
https://jaeger.dev.storj.io/trace/51bf53c7068cf40a
FAIL (2025-12-29 02:40:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ) context deadline exceeded
======LOGS======
gments...
Dec 29 02:39:53.777275456 (563h10m53.648358491s) [upload=493762]: Got next segment. Inline: false
Dec 29 02:39:53.777327304 (563h10m53.648410349s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:39:53.777410803 (563h10m53.648493849s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:39:54.023748009 (563h10m53.894831064s) [seg_pos={0 0}, upload=493762]: Begin upload segment...
Dec 29 02:39:54.023801421 (563h10m53.894884466s) [seg_pos={0 0}, upload=493762]: Done begin upload segment.
Dec 29 02:39:54.023834684 (563h10m53.894917719s) [noise=true , address=77.90.31.21:4880     , piece_id=WOPLT7WKIWNSAUFVKF7UWT6ALRBZVAXWILGAS7GA564GB4ET23DA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.023842238 (563h10m53.894925283s) [noise=true , address=45.91.116.8:4045     , piece_id=CJXHSD7QWZ64IXKM32VJSNS6JJPU4FDPPBCMT4PKU5E5LT7C3OBQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024388709 (563h10m53.895471754s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024673161 (563h10m53.895756206s) [noise=true , address=45.91.116.10:4082    , piece_id=H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.024862290 (563h10m53.895945335s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025154277 (563h10m53.896237312s) [noise=true , address=145.79.227.32:4834   , piece_id=FJ6XR4NWFRSM6P5XVYDQLUIEMUSTTMTO7ELXTUCLM4RJDXNNVCDQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025293843 (563h10m53.896376878s) [noise=true , address=45.91.116.9:4264     , piece_id=BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025492059 (563h10m53.896575105s) [noise=true , address=145.79.227.15:4873   , piece_id=VZEHORCCR7LRA66YDAINKLO64V7SVEX2CBLLXELD3TNI6M6S3XBA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025707550 (563h10m53.896790605s) [noise=true , address=145.79.227.19:4078   , piece_id=2HBNHILFJYFTUHDH4G72TG4AIXX2P2HUVLC62HEJ75MY7JMDL6YQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.025861543 (563h10m53.896944588s) [noise=true , address=145.79.227.47:4265   , piece_id=5PW5RL3AXSJBJCHDWRHT4JVD4CWUMYINDERV6OSQJKFUIGG256YA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026158829 (563h10m53.897241874s) [noise=true , address=145.79.227.51:4409   , piece_id=UX2EYGB2NF45FJREOB2UQZAAVEGGEKGZS5QNVT2R5J47GQMT2HVA, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026390500 (563h10m53.897473545s) [noise=true , address=145.79.227.14:4840   , piece_id=COQ7TGCCQXNUZRA63RUAP4M2HTCUIG2C4XICF5BKWOQ7BPQEKGHQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026573509 (563h10m53.897656544s) [noise=true , address=145.79.227.16:4910   , piece_id=7TYQ4NWXORTLOGNOTPKNC4F2HKGTYTSVPUOMBKW7R5ZXORUDHKEQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.026859102 (563h10m53.897942147s) [noise=true , address=145.79.227.6:4258    , piece_id=ZUE3RX265SY2DPC2NFZTO4T6HQ3EJQCVLQQNRRJVISTULGUX5PRQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.027126732 (563h10m53.898209777s) [noise=true , address=145.79.227.39:4013   , piece_id=TRJBFHI6JYRUJWKRHHYAEG5TORM3FT6S53K4NKSI3YU5WXRXAUVQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:54.027546471 (563h10m53.898629516s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Uploading piece...
Dec 29 02:39:55.749109939 (563h10m55.620192984s) [upload=493762]: Next returned nil segment
Dec 29 02:39:55.749125449 (563h10m55.620208484s) [upload=493762]: Waiting for error group managing segments...
Dec 29 02:39:56.038419480 (563h10m55.909502515s) [noise=true , address=145.79.227.6:4258    , piece_id=ZUE3RX265SY2DPC2NFZTO4T6HQ3EJQCVLQQNRRJVISTULGUX5PRQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.039202140 (563h10m55.910285185s) [noise=true , address=145.79.227.14:4840   , piece_id=COQ7TGCCQXNUZRA63RUAP4M2HTCUIG2C4XICF5BKWOQ7BPQEKGHQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.046728070 (563h10m55.917811105s) [noise=true , address=145.79.227.51:4409   , piece_id=UX2EYGB2NF45FJREOB2UQZAAVEGGEKGZS5QNVT2R5J47GQMT2HVA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.056835877 (563h10m55.927918922s) [noise=true , address=145.79.227.15:4873   , piece_id=VZEHORCCR7LRA66YDAINKLO64V7SVEX2CBLLXELD3TNI6M6S3XBA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.089295214 (563h10m55.960378249s) [noise=true , address=145.79.227.32:4834   , piece_id=FJ6XR4NWFRSM6P5XVYDQLUIEMUSTTMTO7ELXTUCLM4RJDXNNVCDQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.089598632 (563h10m55.960681667s) [noise=true , address=145.79.227.16:4910   , piece_id=7TYQ4NWXORTLOGNOTPKNC4F2HKGTYTSVPUOMBKW7R5ZXORUDHKEQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.113200221 (563h10m55.984283256s) [noise=true , address=145.79.227.19:4078   , piece_id=2HBNHILFJYFTUHDH4G72TG4AIXX2P2HUVLC62HEJ75MY7JMDL6YQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.125066026 (563h10m55.996149071s) [noise=true , address=145.79.227.39:4013   , piece_id=TRJBFHI6JYRUJWKRHHYAEG5TORM3FT6S53K4NKSI3YU5WXRXAUVQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.144873181 (563h10m56.015956216s) [noise=true , address=145.79.227.47:4265   , piece_id=5PW5RL3AXSJBJCHDWRHT4JVD4CWUMYINDERV6OSQJKFUIGG256YA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.280035867 (563h10m56.151118902s) [noise=true , address=45.91.116.9:4264     , piece_id=BB2Y3GLSD7XGSJW2EVAAWE375VGQAIKGDLVZFHU6CZZRWN7AVJUQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.339585787 (563h10m56.210668822s) [noise=true , address=77.90.31.21:4880     , piece_id=WOPLT7WKIWNSAUFVKF7UWT6ALRBZVAXWILGAS7GA564GB4ET23DA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.411543494 (563h10m56.282626529s) [noise=true , address=45.91.116.10:4082    , piece_id=H3VAQJCLUHMKY4IREDS4AYGLBX4Z22ZMIQPGHEIE5E5ZKJOWOKXA, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.419290625 (563h10m56.29037366s) [noise=true , address=45.91.116.8:4045     , piece_id=CJXHSD7QWZ64IXKM32VJSNS6JJPU4FDPPBCMT4PKU5E5LT7C3OBQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: <nil>
Dec 29 02:39:56.419343947 (563h10m56.290426982s) [seg_pos={0 0}, upload=493762]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:39:56.419513470 (563h10m56.290596515s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 02:39:56.419522888 (563h10m56.290605923s) [noise=true , address=145.79.227.35:4481   , piece_id=7BTKVQCLATHPSQ6QNA2LF667ADJPL5AG23G2MIDZPBYG2XHFAF3Q, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419605014 (563h10m56.290688049s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47038->45.91.116.4:4937: use of closed network connection
Dec 29 02:39:56.419619622 (563h10m56.290702657s) [noise=true , address=45.91.116.4:4937     , piece_id=WEYOE5ORXTSOLLXTZSVXDGMBPZRZW32SJIOD5PBX4H2RQKH3ZBYQ, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419793293 (563h10m56.290876328s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49450->77.90.31.10:4520: use of closed network connection
Dec 29 02:39:56.419800396 (563h10m56.290883431s) [noise=true , address=77.90.31.10:4520     , piece_id=E3XTF6SBH3X2OAHX2S73NJI7PDI22GNBUXZ2RG2FPKDF6DBLGE3A, seg_pos={0 0}, upload=493762]: Long tail cancellation due to optimal threshold
Dec 29 02:39:56.419815314 (563h10m56.290898349s) [seg_pos={0 0}, upload=493762]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:39:56.419911388 (563h10m56.290994453s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:39:56.419919242 (563h10m56.291002287s) [upload=493762]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:39:56.724029667 (563h10m56.595112702s) [upload=493762]: Done uploading segments...
Dec 29 02:39:56.724049535 (563h10m56.59513257s) [upload=493762]: Upload finished. err: <nil>
Duration: 30.003940407s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........................................X..................]
duration: 
3.639197818s
trace: 
https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441
FAIL (2025-12-29 02:39:56 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.639252512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.X......X...X....X.....X.........X.X.....XX...XX.XXX..X.X..X]
duration: 
30.003727829s
trace: 
https://jaeger.dev.storj.io/trace/379bd185d9b1c8d5
FAIL (2025-12-29 02:39:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 4LBY7TFFRCMTRDTL2NU2L5SFINMYMLRPA25Q4S2NEHMJ37PU2JRA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: JCK4UEIIIAPZ5ILJBW34N4VQAYGM43K2434IU2BW3Z55SLBKZIIQ) context deadline exceeded
======LOGS======
Dec 29 02:39:23.560504936 (563h10m23.431587981s) [upload=493756]: Starting upload
Dec 29 02:39:23.560573828 (563h10m23.431656893s) [upload=493756]: Done starting upload
Dec 29 02:39:23.560660702 (563h10m23.431743738s) [upload=493756]: Uploading segments...
Dec 29 02:39:23.560750384 (563h10m23.431833419s) [upload=493756]: Got next segment. Inline: false
Dec 29 02:39:23.560818133 (563h10m23.431901168s) [upload=493756]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 02:39:23.560823613 (563h10m23.431906648s) [upload=493756]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 02:39:23.814242373 (563h10m23.685325408s) [seg_pos={0 0}, upload=493756]: Begin upload segment...
Dec 29 02:39:23.814291728 (563h10m23.685374763s) [seg_pos={0 0}, upload=493756]: Done begin upload segment.
Dec 29 02:39:23.814335050 (563h10m23.685418095s) [noise=true , address=145.79.227.15:4873   , piece_id=UH3SLC77TZ2ZIKFOA6WBY3UAXHTNUT6WFMGBLM7PTNIQ6VRFGISA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.814346992 (563h10m23.685430027s) [noise=true , address=45.91.116.9:4263     , piece_id=IFAYE3JFS225FDB6W6DWY65FATL3ILR52I3QEG33PN4HVDBO5YQQ, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.814393250 (563h10m23.685476285s) [noise=true , address=45.91.116.5:4300     , piece_id=ET7MYY5VDPPREDISSFQRZD2W7RVUBI577MD6QN7CU2CAX23BTOPA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.814745291 (563h10m23.685828326s) [noise=true , address=145.79.227.6:4258    , piece_id=IVEFAJ74BON6T5S5NU5SWLFRU5NOM6VM73O2AFB56LWWPWSLJOAA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.814753186 (563h10m23.685836221s) [noise=true , address=45.91.116.12:4225    , piece_id=JCK4UEIIIAPZ5ILJBW34N4VQAYGM43K2434IU2BW3Z55SLBKZIIQ, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.815098925 (563h10m23.68618196s) [noise=true , address=145.79.227.48:4302   , piece_id=KO6B4M32HK4BGEHFI6FDD6XPUIB5XOVCI4PZSIGLAFY4QYTR4TQQ, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.815112610 (563h10m23.686195645s) [noise=true , address=45.91.116.13:4153    , piece_id=4LBY7TFFRCMTRDTL2NU2L5SFINMYMLRPA25Q4S2NEHMJ37PU2JRA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.815375070 (563h10m23.686458106s) [noise=true , address=145.79.227.45:4230   , piece_id=TKJM6LIE6SCAY3IKZTHKRF6EPSVYBHCIUD2SKGI5TV2AJVONUC6Q, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.815416649 (563h10m23.686499685s) [noise=true , address=145.79.227.40:4051   , piece_id=UPCOL6PSNDSQHPA5WI2MU25ORONMXLYVQAUFGQXPUNYCL3CHYPYA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.815698847 (563h10m23.686781882s) [noise=true , address=45.91.116.11:4117    , piece_id=57Y6ZUC47KXI5IXFIIZRFVA4EY32JMQHYC4AWPHLQJWGXROYVWSA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.815708405 (563h10m23.68679144s) [noise=true , address=145.79.227.39:4013   , piece_id=G7HBECSBKRWPSV2AZ3TOMN5D2BJTS2OVNP556U3AIB6YSPBHE2PA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.815937732 (563h10m23.687020767s) [noise=true , address=45.91.116.4:4937     , piece_id=4C7UDD3NQ7WRNVF2QCPWVMWWJWFU2TNFYK57RMKYDX5BB6JLILLA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.816182758 (563h10m23.687265793s) [noise=true , address=77.90.31.11:4556     , piece_id=F2VN33QB624WTNGJEA6ZS4SAB5O7R4JNKYR26NSIHLTWVRKP7GFA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.816709681 (563h10m23.687792717s) [noise=true , address=145.79.227.36:4519   , piece_id=CKATF554ETZXY2YRAIRYHWY7DFBOAVBDCDSTRJXFEJAVUABYVFHQ, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.817332628 (563h10m23.688415663s) [noise=true , address=145.79.227.11:4730   , piece_id=GBEUZ4JQATGJNLPK4RFUFJ322SQHXKXDMREQXWRTHZTSOEIJ3OLQ, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:23.817516287 (563h10m23.688599322s) [noise=true , address=45.91.116.7:4012     , piece_id=CZI5664KFYGG57HOOXSW3S6S4SZ5XPUPOJF3KL64W6IZQYVGYOQA, seg_pos={0 0}, upload=493756]: Uploading piece...
Dec 29 02:39:25.519440515 (563h10m25.39052355s) [upload=493756]: Next returned nil segment
Dec 29 02:39:25.519459801 (563h10m25.390542836s) [upload=493756]: Waiting for error group managing segments...
Dec 29 02:39:25.801427183 (563h10m25.672510218s) [noise=true , address=145.79.227.6:4258    , piece_id=IVEFAJ74BON6T5S5NU5SWLFRU5NOM6VM73O2AFB56LWWPWSLJOAA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:25.813686335 (563h10m25.68476937s) [noise=true , address=145.79.227.15:4873   , piece_id=UH3SLC77TZ2ZIKFOA6WBY3UAXHTNUT6WFMGBLM7PTNIQ6VRFGISA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:25.817532769 (563h10m25.688615804s) [noise=true , address=145.79.227.48:4302   , piece_id=KO6B4M32HK4BGEHFI6FDD6XPUIB5XOVCI4PZSIGLAFY4QYTR4TQQ, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:25.823909181 (563h10m25.694992226s) [noise=true , address=145.79.227.45:4230   , piece_id=TKJM6LIE6SCAY3IKZTHKRF6EPSVYBHCIUD2SKGI5TV2AJVONUC6Q, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:25.835232562 (563h10m25.706315597s) [noise=true , address=145.79.227.11:4730   , piece_id=GBEUZ4JQATGJNLPK4RFUFJ322SQHXKXDMREQXWRTHZTSOEIJ3OLQ, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:25.840334006 (563h10m25.711417041s) [noise=true , address=145.79.227.39:4013   , piece_id=G7HBECSBKRWPSV2AZ3TOMN5D2BJTS2OVNP556U3AIB6YSPBHE2PA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:25.842614327 (563h10m25.713697372s) [noise=true , address=145.79.227.36:4519   , piece_id=CKATF554ETZXY2YRAIRYHWY7DFBOAVBDCDSTRJXFEJAVUABYVFHQ, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:26.030326955 (563h10m25.90140999s) [noise=true , address=45.91.116.5:4300     , piece_id=ET7MYY5VDPPREDISSFQRZD2W7RVUBI577MD6QN7CU2CAX23BTOPA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:26.050830777 (563h10m25.921913812s) [noise=true , address=45.91.116.7:4012     , piece_id=CZI5664KFYGG57HOOXSW3S6S4SZ5XPUPOJF3KL64W6IZQYVGYOQA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:26.062280088 (563h10m25.933363134s) [noise=true , address=77.90.31.11:4556     , piece_id=F2VN33QB624WTNGJEA6ZS4SAB5O7R4JNKYR26NSIHLTWVRKP7GFA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:26.070075391 (563h10m25.941158426s) [noise=true , address=45.91.116.13:4153    , piece_id=4LBY7TFFRCMTRDTL2NU2L5SFINMYMLRPA25Q4S2NEHMJ37PU2JRA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:26.175567882 (563h10m26.046650937s) [noise=true , address=45.91.116.9:4263     , piece_id=IFAYE3JFS225FDB6W6DWY65FATL3ILR52I3QEG33PN4HVDBO5YQQ, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:26.766297079 (563h10m26.637380114s) [noise=true , address=45.91.116.12:4225    , piece_id=JCK4UEIIIAPZ5ILJBW34N4VQAYGM43K2434IU2BW3Z55SLBKZIIQ, seg_pos={0 0}, upload=493756]: Done uploading piece. err: <nil>
Dec 29 02:39:26.766322787 (563h10m26.637405833s) [seg_pos={0 0}, upload=493756]: Segment reached optimal threshold of 13 pieces.
Dec 29 02:39:26.766667875 (563h10m26.63775092s) [noise=true , address=145.79.227.40:4051   , piece_id=UPCOL6PSNDSQHPA5WI2MU25ORONMXLYVQAUFGQXPUNYCL3CHYPYA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 29 02:39:26.766677183 (563h10m26.637760218s) [noise=true , address=145.79.227.40:4051   , piece_id=UPCOL6PSNDSQHPA5WI2MU25ORONMXLYVQAUFGQXPUNYCL3CHYPYA, seg_pos={0 0}, upload=493756]: Long tail cancellation due to optimal threshold
Dec 29 02:39:26.767009014 (563h10m26.63809205s) [noise=true , address=45.91.116.11:4117    , piece_id=57Y6ZUC47KXI5IXFIIZRFVA4EY32JMQHYC4AWPHLQJWGXROYVWSA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 29 02:39:26.767016128 (563h10m26.638099173s) [noise=true , address=45.91.116.11:4117    , piece_id=57Y6ZUC47KXI5IXFIIZRFVA4EY32JMQHYC4AWPHLQJWGXROYVWSA, seg_pos={0 0}, upload=493756]: Long tail cancellation due to optimal threshold
Dec 29 02:39:26.767189478 (563h10m26.638272543s) [noise=true , address=45.91.116.4:4937     , piece_id=4C7UDD3NQ7WRNVF2QCPWVMWWJWFU2TNFYK57RMKYDX5BB6JLILLA, seg_pos={0 0}, upload=493756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 29 02:39:26.767198776 (563h10m26.638281821s) [noise=true , address=45.91.116.4:4937     , piece_id=4C7UDD3NQ7WRNVF2QCPWVMWWJWFU2TNFYK57RMKYDX5BB6JLILLA, seg_pos={0 0}, upload=493756]: Long tail cancellation due to optimal threshold
Dec 29 02:39:26.767220918 (563h10m26.638303963s) [seg_pos={0 0}, upload=493756]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 02:39:26.767302674 (563h10m26.638385709s) [upload=493756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 02:39:26.767308696 (563h10m26.638391741s) [upload=493756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 02:39:27.061522193 (563h10m26.932605248s) [upload=493756]: Done uploading segments...
Dec 29 02:39:27.061546920 (563h10m26.932629965s) [upload=493756]: Upload finished. err: <nil>
Duration: 30.003751104s core uplink us-select-1