Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 906466 reports (Page 11259 of 18130)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
6 out of 60 is failed (threshold=6) [.....X..X..X..................X.......................X....X]
duration: 
5.002810664s
trace: 
https://jaeger.dev.storj.io/trace/5644e5a33aaf27bb
FAIL (2025-12-30 08:42:02 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002830163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
5 out of 60 is failed (threshold=6) [......X..X..X..................X.......................X....]
duration: 
5.001098358s
trace: 
https://jaeger.dev.storj.io/trace/5bb2358f8fab9298
FAIL (2025-12-30 08:41:30 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001177981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [..X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.X.X.X..]
duration: 
5.739290558s
trace: 
https://jaeger.dev.storj.io/trace/7dddfd383e746bd5
FAIL (2025-12-30 08:41:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q) context deadline exceeded
======LOGS======
Dec 30 08:39:53.387296002 (593h10m53.258379037s) [upload=520021]: Starting upload
Dec 30 08:39:53.387357800 (593h10m53.258440845s) [upload=520021]: Done starting upload
Dec 30 08:39:53.387648113 (593h10m53.258731159s) [upload=520021]: Uploading segments...
Dec 30 08:39:53.387696565 (593h10m53.25877961s) [upload=520021]: Got next segment. Inline: false
Dec 30 08:39:53.387737213 (593h10m53.258820258s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:39:53.387743274 (593h10m53.258826309s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:39:53.635887442 (593h10m53.506970487s) [seg_pos={0 0}, upload=520021]: Begin upload segment...
Dec 30 08:39:53.635934943 (593h10m53.507017988s) [seg_pos={0 0}, upload=520021]: Done begin upload segment.
Dec 30 08:39:53.635972985 (593h10m53.50705603s) [noise=true , address=77.90.31.15:4700     , piece_id=6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.636357187 (593h10m53.507440232s) [noise=true , address=145.79.227.18:4006   , piece_id=MZ57NVOKBJNN2PASPWPVJIGEZNPNRJNCRRDKHBGYK4QYFOQCAPCA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.636830397 (593h10m53.507913442s) [noise=true , address=145.79.227.11:4730   , piece_id=AXMW5WSQDUKK747NPWAJOT6RNLC2LVB6UKQI2ZV5MEEEZXUAG7LA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637130108 (593h10m53.508213154s) [noise=true , address=145.79.227.15:4876   , piece_id=AC6QWQJ5ZCOHG2PS7VY3LTAKTBV6NYDZLXJTIJHXZNCWH2I6PXTQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637462762 (593h10m53.508545797s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637759086 (593h10m53.508842121s) [noise=true , address=145.79.227.50:4374   , piece_id=ZXLAJOLP3HMEWAPFGDJGVA2AGDKESIOD4UORWZURKDCL2ILI7BDA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.638069117 (593h10m53.509152162s) [noise=true , address=145.79.227.17:4972   , piece_id=EPBFVA7VZSKWGQNEPLE7Q7WTNSBZ6VMRQT2GGYDDNTGMPXFWH3XA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.638336396 (593h10m53.509419431s) [noise=true , address=145.79.227.38:4978   , piece_id=KA5VWBVKVAAM47ZW5HAWGFETVBQJZ7DCXX245BFZWTTNF4FFROXA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.640625876 (593h10m53.511708911s) [noise=true , address=145.79.227.48:4301   , piece_id=P225PSRJU4XVOD6KUM5DSNAJW2QKEBDZKQ4HK2IPTCAAHPM5XJNQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.641977519 (593h10m53.513060564s) [noise=true , address=77.90.31.18:4808     , piece_id=QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.643205457 (593h10m53.514288503s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.644506715 (593h10m53.51558975s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.645691180 (593h10m53.516774215s) [noise=true , address=145.79.227.27:4689   , piece_id=V34UKFC33PVX2HU4I3O2QPXTT5YCNBAADHWZ2N3OCKV7C2LCNS2Q, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.646945278 (593h10m53.518028323s) [noise=true , address=145.79.227.21:4475   , piece_id=RWIS7EO7AQUNPC7EHI36QXVQUL4LAEXZA7V3LRBPOF2SUAI7ZIIA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.648513675 (593h10m53.51959671s) [noise=true , address=145.79.227.16:4910   , piece_id=CZ3EVO24BLXU7WFJDMXYRMOKMUZG4IM7RTFNQEQIFHHRLADOY5KA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.649941413 (593h10m53.521024448s) [noise=true , address=145.79.227.51:4409   , piece_id=RBDHSRJ7S4L57UVEWYAX3VRRLMEN7UXD627AACEE5U7OAO5RAE4A, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:55.259217631 (593h10m55.130300666s) [upload=520021]: Next returned nil segment
Dec 30 08:39:55.259262366 (593h10m55.130345401s) [upload=520021]: Waiting for error group managing segments...
Dec 30 08:39:55.616705737 (593h10m55.487788782s) [noise=true , address=145.79.227.38:4978   , piece_id=KA5VWBVKVAAM47ZW5HAWGFETVBQJZ7DCXX245BFZWTTNF4FFROXA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.622017452 (593h10m55.493100487s) [noise=true , address=145.79.227.11:4730   , piece_id=AXMW5WSQDUKK747NPWAJOT6RNLC2LVB6UKQI2ZV5MEEEZXUAG7LA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.626833022 (593h10m55.497916067s) [noise=true , address=145.79.227.50:4374   , piece_id=ZXLAJOLP3HMEWAPFGDJGVA2AGDKESIOD4UORWZURKDCL2ILI7BDA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.637234768 (593h10m55.508317804s) [noise=true , address=145.79.227.17:4972   , piece_id=EPBFVA7VZSKWGQNEPLE7Q7WTNSBZ6VMRQT2GGYDDNTGMPXFWH3XA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.641238923 (593h10m55.512321958s) [noise=true , address=145.79.227.16:4910   , piece_id=CZ3EVO24BLXU7WFJDMXYRMOKMUZG4IM7RTFNQEQIFHHRLADOY5KA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.645453077 (593h10m55.516536113s) [noise=true , address=145.79.227.51:4409   , piece_id=RBDHSRJ7S4L57UVEWYAX3VRRLMEN7UXD627AACEE5U7OAO5RAE4A, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.650785110 (593h10m55.521868145s) [noise=true , address=145.79.227.15:4876   , piece_id=AC6QWQJ5ZCOHG2PS7VY3LTAKTBV6NYDZLXJTIJHXZNCWH2I6PXTQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.662595559 (593h10m55.533678604s) [noise=true , address=145.79.227.27:4689   , piece_id=V34UKFC33PVX2HU4I3O2QPXTT5YCNBAADHWZ2N3OCKV7C2LCNS2Q, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.675653524 (593h10m55.546736569s) [noise=true , address=145.79.227.48:4301   , piece_id=P225PSRJU4XVOD6KUM5DSNAJW2QKEBDZKQ4HK2IPTCAAHPM5XJNQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.707941956 (593h10m55.579024991s) [noise=true , address=145.79.227.18:4006   , piece_id=MZ57NVOKBJNN2PASPWPVJIGEZNPNRJNCRRDKHBGYK4QYFOQCAPCA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.721419040 (593h10m55.592502085s) [noise=true , address=145.79.227.21:4475   , piece_id=RWIS7EO7AQUNPC7EHI36QXVQUL4LAEXZA7V3LRBPOF2SUAI7ZIIA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.802194328 (593h10m55.673277373s) [noise=true , address=77.90.31.18:4808     , piece_id=QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.865677420 (593h10m55.736760465s) [noise=true , address=77.90.31.15:4700     , piece_id=6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.865709221 (593h10m55.736792266s) [seg_pos={0 0}, upload=520021]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:39:55.866116567 (593h10m55.737199602s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866158296 (593h10m55.737241331s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866206709 (593h10m55.737289764s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866218251 (593h10m55.737301286s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866496661 (593h10m55.737579706s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866590850 (593h10m55.737673926s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866619164 (593h10m55.737702199s) [seg_pos={0 0}, upload=520021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:39:55.866796252 (593h10m55.737879297s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:39:55.866805108 (593h10m55.737888153s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:39:56.161180828 (593h10m56.032263863s) [upload=520021]: Done uploading segments...
Dec 30 08:39:56.161201167 (593h10m56.032284202s) [upload=520021]: Upload finished. err: <nil>
Duration: 5.73931746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [X................X...............................X..........]
duration: 
37.874162403s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:41:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 37.87420257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
5 out of 60 is failed (threshold=6) [.......X..X..X..................X.......................X...]
duration: 
5.001163022s
trace: 
https://jaeger.dev.storj.io/trace/5bb2358f8fab9298
FAIL (2025-12-30 08:41:07 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001196647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.X.X.X.]
duration: 
5.72729063s
trace: 
https://jaeger.dev.storj.io/trace/7dddfd383e746bd5
FAIL (2025-12-30 08:40:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q) context deadline exceeded
======LOGS======
Dec 30 08:39:53.387296002 (593h10m53.258379037s) [upload=520021]: Starting upload
Dec 30 08:39:53.387357800 (593h10m53.258440845s) [upload=520021]: Done starting upload
Dec 30 08:39:53.387648113 (593h10m53.258731159s) [upload=520021]: Uploading segments...
Dec 30 08:39:53.387696565 (593h10m53.25877961s) [upload=520021]: Got next segment. Inline: false
Dec 30 08:39:53.387737213 (593h10m53.258820258s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:39:53.387743274 (593h10m53.258826309s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:39:53.635887442 (593h10m53.506970487s) [seg_pos={0 0}, upload=520021]: Begin upload segment...
Dec 30 08:39:53.635934943 (593h10m53.507017988s) [seg_pos={0 0}, upload=520021]: Done begin upload segment.
Dec 30 08:39:53.635972985 (593h10m53.50705603s) [noise=true , address=77.90.31.15:4700     , piece_id=6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.636357187 (593h10m53.507440232s) [noise=true , address=145.79.227.18:4006   , piece_id=MZ57NVOKBJNN2PASPWPVJIGEZNPNRJNCRRDKHBGYK4QYFOQCAPCA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.636830397 (593h10m53.507913442s) [noise=true , address=145.79.227.11:4730   , piece_id=AXMW5WSQDUKK747NPWAJOT6RNLC2LVB6UKQI2ZV5MEEEZXUAG7LA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637130108 (593h10m53.508213154s) [noise=true , address=145.79.227.15:4876   , piece_id=AC6QWQJ5ZCOHG2PS7VY3LTAKTBV6NYDZLXJTIJHXZNCWH2I6PXTQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637462762 (593h10m53.508545797s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637759086 (593h10m53.508842121s) [noise=true , address=145.79.227.50:4374   , piece_id=ZXLAJOLP3HMEWAPFGDJGVA2AGDKESIOD4UORWZURKDCL2ILI7BDA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.638069117 (593h10m53.509152162s) [noise=true , address=145.79.227.17:4972   , piece_id=EPBFVA7VZSKWGQNEPLE7Q7WTNSBZ6VMRQT2GGYDDNTGMPXFWH3XA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.638336396 (593h10m53.509419431s) [noise=true , address=145.79.227.38:4978   , piece_id=KA5VWBVKVAAM47ZW5HAWGFETVBQJZ7DCXX245BFZWTTNF4FFROXA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.640625876 (593h10m53.511708911s) [noise=true , address=145.79.227.48:4301   , piece_id=P225PSRJU4XVOD6KUM5DSNAJW2QKEBDZKQ4HK2IPTCAAHPM5XJNQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.641977519 (593h10m53.513060564s) [noise=true , address=77.90.31.18:4808     , piece_id=QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.643205457 (593h10m53.514288503s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.644506715 (593h10m53.51558975s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.645691180 (593h10m53.516774215s) [noise=true , address=145.79.227.27:4689   , piece_id=V34UKFC33PVX2HU4I3O2QPXTT5YCNBAADHWZ2N3OCKV7C2LCNS2Q, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.646945278 (593h10m53.518028323s) [noise=true , address=145.79.227.21:4475   , piece_id=RWIS7EO7AQUNPC7EHI36QXVQUL4LAEXZA7V3LRBPOF2SUAI7ZIIA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.648513675 (593h10m53.51959671s) [noise=true , address=145.79.227.16:4910   , piece_id=CZ3EVO24BLXU7WFJDMXYRMOKMUZG4IM7RTFNQEQIFHHRLADOY5KA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.649941413 (593h10m53.521024448s) [noise=true , address=145.79.227.51:4409   , piece_id=RBDHSRJ7S4L57UVEWYAX3VRRLMEN7UXD627AACEE5U7OAO5RAE4A, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:55.259217631 (593h10m55.130300666s) [upload=520021]: Next returned nil segment
Dec 30 08:39:55.259262366 (593h10m55.130345401s) [upload=520021]: Waiting for error group managing segments...
Dec 30 08:39:55.616705737 (593h10m55.487788782s) [noise=true , address=145.79.227.38:4978   , piece_id=KA5VWBVKVAAM47ZW5HAWGFETVBQJZ7DCXX245BFZWTTNF4FFROXA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.622017452 (593h10m55.493100487s) [noise=true , address=145.79.227.11:4730   , piece_id=AXMW5WSQDUKK747NPWAJOT6RNLC2LVB6UKQI2ZV5MEEEZXUAG7LA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.626833022 (593h10m55.497916067s) [noise=true , address=145.79.227.50:4374   , piece_id=ZXLAJOLP3HMEWAPFGDJGVA2AGDKESIOD4UORWZURKDCL2ILI7BDA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.637234768 (593h10m55.508317804s) [noise=true , address=145.79.227.17:4972   , piece_id=EPBFVA7VZSKWGQNEPLE7Q7WTNSBZ6VMRQT2GGYDDNTGMPXFWH3XA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.641238923 (593h10m55.512321958s) [noise=true , address=145.79.227.16:4910   , piece_id=CZ3EVO24BLXU7WFJDMXYRMOKMUZG4IM7RTFNQEQIFHHRLADOY5KA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.645453077 (593h10m55.516536113s) [noise=true , address=145.79.227.51:4409   , piece_id=RBDHSRJ7S4L57UVEWYAX3VRRLMEN7UXD627AACEE5U7OAO5RAE4A, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.650785110 (593h10m55.521868145s) [noise=true , address=145.79.227.15:4876   , piece_id=AC6QWQJ5ZCOHG2PS7VY3LTAKTBV6NYDZLXJTIJHXZNCWH2I6PXTQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.662595559 (593h10m55.533678604s) [noise=true , address=145.79.227.27:4689   , piece_id=V34UKFC33PVX2HU4I3O2QPXTT5YCNBAADHWZ2N3OCKV7C2LCNS2Q, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.675653524 (593h10m55.546736569s) [noise=true , address=145.79.227.48:4301   , piece_id=P225PSRJU4XVOD6KUM5DSNAJW2QKEBDZKQ4HK2IPTCAAHPM5XJNQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.707941956 (593h10m55.579024991s) [noise=true , address=145.79.227.18:4006   , piece_id=MZ57NVOKBJNN2PASPWPVJIGEZNPNRJNCRRDKHBGYK4QYFOQCAPCA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.721419040 (593h10m55.592502085s) [noise=true , address=145.79.227.21:4475   , piece_id=RWIS7EO7AQUNPC7EHI36QXVQUL4LAEXZA7V3LRBPOF2SUAI7ZIIA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.802194328 (593h10m55.673277373s) [noise=true , address=77.90.31.18:4808     , piece_id=QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.865677420 (593h10m55.736760465s) [noise=true , address=77.90.31.15:4700     , piece_id=6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.865709221 (593h10m55.736792266s) [seg_pos={0 0}, upload=520021]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:39:55.866116567 (593h10m55.737199602s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866158296 (593h10m55.737241331s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866206709 (593h10m55.737289764s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866218251 (593h10m55.737301286s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866496661 (593h10m55.737579706s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866590850 (593h10m55.737673926s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866619164 (593h10m55.737702199s) [seg_pos={0 0}, upload=520021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:39:55.866796252 (593h10m55.737879297s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:39:55.866805108 (593h10m55.737888153s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:39:56.161180828 (593h10m56.032263863s) [upload=520021]: Done uploading segments...
Dec 30 08:39:56.161201167 (593h10m56.032284202s) [upload=520021]: Upload finished. err: <nil>
Duration: 5.727308594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.X.X.X]
duration: 
30.003251181s
trace: 
https://jaeger.dev.storj.io/trace/7dddfd383e746bd5
FAIL (2025-12-30 08:40:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q) context deadline exceeded
======LOGS======
Dec 30 08:39:53.387296002 (593h10m53.258379037s) [upload=520021]: Starting upload
Dec 30 08:39:53.387357800 (593h10m53.258440845s) [upload=520021]: Done starting upload
Dec 30 08:39:53.387648113 (593h10m53.258731159s) [upload=520021]: Uploading segments...
Dec 30 08:39:53.387696565 (593h10m53.25877961s) [upload=520021]: Got next segment. Inline: false
Dec 30 08:39:53.387737213 (593h10m53.258820258s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:39:53.387743274 (593h10m53.258826309s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:39:53.635887442 (593h10m53.506970487s) [seg_pos={0 0}, upload=520021]: Begin upload segment...
Dec 30 08:39:53.635934943 (593h10m53.507017988s) [seg_pos={0 0}, upload=520021]: Done begin upload segment.
Dec 30 08:39:53.635972985 (593h10m53.50705603s) [noise=true , address=77.90.31.15:4700     , piece_id=6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.636357187 (593h10m53.507440232s) [noise=true , address=145.79.227.18:4006   , piece_id=MZ57NVOKBJNN2PASPWPVJIGEZNPNRJNCRRDKHBGYK4QYFOQCAPCA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.636830397 (593h10m53.507913442s) [noise=true , address=145.79.227.11:4730   , piece_id=AXMW5WSQDUKK747NPWAJOT6RNLC2LVB6UKQI2ZV5MEEEZXUAG7LA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637130108 (593h10m53.508213154s) [noise=true , address=145.79.227.15:4876   , piece_id=AC6QWQJ5ZCOHG2PS7VY3LTAKTBV6NYDZLXJTIJHXZNCWH2I6PXTQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637462762 (593h10m53.508545797s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.637759086 (593h10m53.508842121s) [noise=true , address=145.79.227.50:4374   , piece_id=ZXLAJOLP3HMEWAPFGDJGVA2AGDKESIOD4UORWZURKDCL2ILI7BDA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.638069117 (593h10m53.509152162s) [noise=true , address=145.79.227.17:4972   , piece_id=EPBFVA7VZSKWGQNEPLE7Q7WTNSBZ6VMRQT2GGYDDNTGMPXFWH3XA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.638336396 (593h10m53.509419431s) [noise=true , address=145.79.227.38:4978   , piece_id=KA5VWBVKVAAM47ZW5HAWGFETVBQJZ7DCXX245BFZWTTNF4FFROXA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.640625876 (593h10m53.511708911s) [noise=true , address=145.79.227.48:4301   , piece_id=P225PSRJU4XVOD6KUM5DSNAJW2QKEBDZKQ4HK2IPTCAAHPM5XJNQ, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.641977519 (593h10m53.513060564s) [noise=true , address=77.90.31.18:4808     , piece_id=QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.643205457 (593h10m53.514288503s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.644506715 (593h10m53.51558975s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.645691180 (593h10m53.516774215s) [noise=true , address=145.79.227.27:4689   , piece_id=V34UKFC33PVX2HU4I3O2QPXTT5YCNBAADHWZ2N3OCKV7C2LCNS2Q, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.646945278 (593h10m53.518028323s) [noise=true , address=145.79.227.21:4475   , piece_id=RWIS7EO7AQUNPC7EHI36QXVQUL4LAEXZA7V3LRBPOF2SUAI7ZIIA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.648513675 (593h10m53.51959671s) [noise=true , address=145.79.227.16:4910   , piece_id=CZ3EVO24BLXU7WFJDMXYRMOKMUZG4IM7RTFNQEQIFHHRLADOY5KA, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:53.649941413 (593h10m53.521024448s) [noise=true , address=145.79.227.51:4409   , piece_id=RBDHSRJ7S4L57UVEWYAX3VRRLMEN7UXD627AACEE5U7OAO5RAE4A, seg_pos={0 0}, upload=520021]: Uploading piece...
Dec 30 08:39:55.259217631 (593h10m55.130300666s) [upload=520021]: Next returned nil segment
Dec 30 08:39:55.259262366 (593h10m55.130345401s) [upload=520021]: Waiting for error group managing segments...
Dec 30 08:39:55.616705737 (593h10m55.487788782s) [noise=true , address=145.79.227.38:4978   , piece_id=KA5VWBVKVAAM47ZW5HAWGFETVBQJZ7DCXX245BFZWTTNF4FFROXA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.622017452 (593h10m55.493100487s) [noise=true , address=145.79.227.11:4730   , piece_id=AXMW5WSQDUKK747NPWAJOT6RNLC2LVB6UKQI2ZV5MEEEZXUAG7LA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.626833022 (593h10m55.497916067s) [noise=true , address=145.79.227.50:4374   , piece_id=ZXLAJOLP3HMEWAPFGDJGVA2AGDKESIOD4UORWZURKDCL2ILI7BDA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.637234768 (593h10m55.508317804s) [noise=true , address=145.79.227.17:4972   , piece_id=EPBFVA7VZSKWGQNEPLE7Q7WTNSBZ6VMRQT2GGYDDNTGMPXFWH3XA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.641238923 (593h10m55.512321958s) [noise=true , address=145.79.227.16:4910   , piece_id=CZ3EVO24BLXU7WFJDMXYRMOKMUZG4IM7RTFNQEQIFHHRLADOY5KA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.645453077 (593h10m55.516536113s) [noise=true , address=145.79.227.51:4409   , piece_id=RBDHSRJ7S4L57UVEWYAX3VRRLMEN7UXD627AACEE5U7OAO5RAE4A, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.650785110 (593h10m55.521868145s) [noise=true , address=145.79.227.15:4876   , piece_id=AC6QWQJ5ZCOHG2PS7VY3LTAKTBV6NYDZLXJTIJHXZNCWH2I6PXTQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.662595559 (593h10m55.533678604s) [noise=true , address=145.79.227.27:4689   , piece_id=V34UKFC33PVX2HU4I3O2QPXTT5YCNBAADHWZ2N3OCKV7C2LCNS2Q, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.675653524 (593h10m55.546736569s) [noise=true , address=145.79.227.48:4301   , piece_id=P225PSRJU4XVOD6KUM5DSNAJW2QKEBDZKQ4HK2IPTCAAHPM5XJNQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.707941956 (593h10m55.579024991s) [noise=true , address=145.79.227.18:4006   , piece_id=MZ57NVOKBJNN2PASPWPVJIGEZNPNRJNCRRDKHBGYK4QYFOQCAPCA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.721419040 (593h10m55.592502085s) [noise=true , address=145.79.227.21:4475   , piece_id=RWIS7EO7AQUNPC7EHI36QXVQUL4LAEXZA7V3LRBPOF2SUAI7ZIIA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.802194328 (593h10m55.673277373s) [noise=true , address=77.90.31.18:4808     , piece_id=QNWYPYWSLP6BQVSPQESFHGI25FINAUZ3FWWYVD32EISGPQJFZMMA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.865677420 (593h10m55.736760465s) [noise=true , address=77.90.31.15:4700     , piece_id=6ZEBA7SDTGP2PMPKDWDBLSNLALNHYYNCA4O3AMXODPHCTCOJSU6Q, seg_pos={0 0}, upload=520021]: Done uploading piece. err: <nil>
Dec 30 08:39:55.865709221 (593h10m55.736792266s) [seg_pos={0 0}, upload=520021]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:39:55.866116567 (593h10m55.737199602s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866158296 (593h10m55.737241331s) [noise=true , address=77.90.31.14:4663     , piece_id=2OHV4NEPTC6C65BWH5ZZDENG2CHJTB3LCUDOTOFFAPBXUS4Q6PAQ, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866206709 (593h10m55.737289764s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866218251 (593h10m55.737301286s) [noise=true , address=77.90.31.11:4554     , piece_id=SRQBRY72X4SHVYY5ODLAZ7RZKFIF44O37BLDAF4VKIP4Z6FCVJZA, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866496661 (593h10m55.737579706s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 30 08:39:55.866590850 (593h10m55.737673926s) [noise=true , address=45.91.116.12:4226    , piece_id=BLCJ4LDQHMDSMK3FRXYAOTAPM4J33PZJ23DW6FN2P5BANTBDOKUA, seg_pos={0 0}, upload=520021]: Long tail cancellation due to optimal threshold
Dec 30 08:39:55.866619164 (593h10m55.737702199s) [seg_pos={0 0}, upload=520021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:39:55.866796252 (593h10m55.737879297s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:39:55.866805108 (593h10m55.737888153s) [upload=520021]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:39:56.161180828 (593h10m56.032263863s) [upload=520021]: Done uploading segments...
Dec 30 08:39:56.161201167 (593h10m56.032284202s) [upload=520021]: Upload finished. err: <nil>
Duration: 30.003270537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.X................X...............................X.........]
duration: 
5.771301001s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:40:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 5.7713217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
6 out of 60 is failed (threshold=6) [X.......X..X..X..................X.......................X..]
duration: 
3.675530647s
trace: 
https://jaeger.dev.storj.io/trace/5bb2358f8fab9298
FAIL (2025-12-30 08:40:08 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.675576264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..X................X...............................X........]
duration: 
41.756851305s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:40:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 41.756883566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.X.X.]
duration: 
5.696096069s
trace: 
https://jaeger.dev.storj.io/trace/23b49ebff208b3b6
FAIL (2025-12-30 08:39:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: QKDQ2GGGXEKGQFB6H6YQ2TWZIHWIBKAUPLNKQRPZ4FKXPHXMMGQQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: YF5IBB3QE3MTR6OQ5DAGSMVDYHST3CCFGGE3DI24OH654XBNBKHQ) context deadline exceeded
======LOGS======
Dec 30 08:38:52.544991460 (593h9m52.416074495s) [upload=520005]: Starting upload
Dec 30 08:38:52.545073707 (593h9m52.416156742s) [upload=520005]: Done starting upload
Dec 30 08:38:52.545177675 (593h9m52.41626072s) [upload=520005]: Uploading segments...
Dec 30 08:38:52.545339312 (593h9m52.416422357s) [upload=520005]: Got next segment. Inline: false
Dec 30 08:38:52.545414255 (593h9m52.41649729s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:38:52.545421569 (593h9m52.416504614s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:38:52.821369549 (593h9m52.692452584s) [seg_pos={0 0}, upload=520005]: Begin upload segment...
Dec 30 08:38:52.821415245 (593h9m52.692498281s) [seg_pos={0 0}, upload=520005]: Done begin upload segment.
Dec 30 08:38:52.821441636 (593h9m52.692524671s) [noise=true , address=145.79.227.42:4121   , piece_id=7YIOKDFQ2YO3IIPOVNZNYQI4UVLREEJTYIN5DANIBRDPDJ2ZLM6A, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.821447878 (593h9m52.692530923s) [noise=true , address=145.79.227.10:4401   , piece_id=Y3NCRNKL4RL7U56GXMBM5KFN4NMW3UZDRRS663ASPEOOKA77Z5TA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.822332231 (593h9m52.693415276s) [noise=true , address=45.91.116.13:4154    , piece_id=RGZ4GXMIUNBHC7CY5ACSJNBEPA6Z7ET7BLPN46L6LSO67GCD5G4Q, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.822775375 (593h9m52.69385842s) [noise=true , address=145.79.227.17:4970   , piece_id=VRYMFHFWHY3XF7S2V2MKFMEVAQUVYWID7QX76ZJX5X4EL5XTVBIQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.823191217 (593h9m52.694274262s) [noise=true , address=145.79.227.3:4151    , piece_id=7LRODGTBCS24MZGIIC2EO57DHLOUSAMDFGCCN2YBBIMCJDQBMQRQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.823662315 (593h9m52.69474536s) [noise=true , address=145.79.227.11:4729   , piece_id=FJ34524DVCHVF53RGXHOGZONGIA5MVMH4TCT7HSYVZAAPELCHK4Q, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.824037969 (593h9m52.695121014s) [noise=true , address=145.79.227.26:4654   , piece_id=BW4T4E5IUD23JHUYFWK5SPHIYGVTNYONRUELBISVOBWBOIMXIBKA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.824446367 (593h9m52.695529412s) [noise=true , address=77.90.31.11:4553     , piece_id=QKDQ2GGGXEKGQFB6H6YQ2TWZIHWIBKAUPLNKQRPZ4FKXPHXMMGQQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.824839165 (593h9m52.6959222s) [noise=true , address=45.91.116.12:4226    , piece_id=XGPB2RECAZXGPH5ZGYY65G4QKADGEJ77F6HK264FZ3DXMT5H32GA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.825242714 (593h9m52.696325759s) [noise=true , address=145.79.227.52:4446   , piece_id=WJ6ZVNDN64YEW3SYLYLVXDSX3LUFDBSFNI4DIH3L4JUEPWF7QIGA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.825636814 (593h9m52.696719849s) [noise=true , address=145.79.227.25:4617   , piece_id=6NLXI4HJ5JFBLOEW6VLE2JK7GOWUZT7RVHLSCTDNJSRFFGMV2NPQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.826131767 (593h9m52.697214812s) [noise=true , address=77.90.31.14:4664     , piece_id=YF5IBB3QE3MTR6OQ5DAGSMVDYHST3CCFGGE3DI24OH654XBNBKHQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.826531267 (593h9m52.697614312s) [noise=true , address=145.79.227.7:4293    , piece_id=NM27NO2Z3SXLC4U5JWPAERLRZIZFIALKXYNR5QFLI54BGAKR63MQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.826968951 (593h9m52.698051996s) [noise=true , address=145.79.227.19:4077   , piece_id=YPROJJC4PXC6DDTW6BGYXIUQSXAY74G73BOP7DNEGKPSO46YM4PA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.827500192 (593h9m52.698583238s) [noise=true , address=45.91.116.11:4117    , piece_id=OGCLJCCITRRPMWUSWXGEKKK6Y7RXIZ4ALTPVNI7DZGQ7VYGKKKWQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.827983984 (593h9m52.699067029s) [noise=true , address=145.79.227.9:4367    , piece_id=ABA6LT3Z27LBIME27DFTMR3NY7GX7WR533ETWUVGRDFLURC4CHMQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:54.532314612 (593h9m54.403397647s) [upload=520005]: Next returned nil segment
Dec 30 08:38:54.532331504 (593h9m54.403414539s) [upload=520005]: Waiting for error group managing segments...
Dec 30 08:38:54.813554008 (593h9m54.684637043s) [noise=true , address=145.79.227.10:4401   , piece_id=Y3NCRNKL4RL7U56GXMBM5KFN4NMW3UZDRRS663ASPEOOKA77Z5TA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.816872887 (593h9m54.687955932s) [noise=true , address=145.79.227.19:4077   , piece_id=YPROJJC4PXC6DDTW6BGYXIUQSXAY74G73BOP7DNEGKPSO46YM4PA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.823199855 (593h9m54.6942829s) [noise=true , address=145.79.227.17:4970   , piece_id=VRYMFHFWHY3XF7S2V2MKFMEVAQUVYWID7QX76ZJX5X4EL5XTVBIQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.824758714 (593h9m54.695841759s) [noise=true , address=145.79.227.11:4729   , piece_id=FJ34524DVCHVF53RGXHOGZONGIA5MVMH4TCT7HSYVZAAPELCHK4Q, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.834126101 (593h9m54.705209136s) [noise=true , address=145.79.227.3:4151    , piece_id=7LRODGTBCS24MZGIIC2EO57DHLOUSAMDFGCCN2YBBIMCJDQBMQRQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.836052599 (593h9m54.707135644s) [noise=true , address=145.79.227.42:4121   , piece_id=7YIOKDFQ2YO3IIPOVNZNYQI4UVLREEJTYIN5DANIBRDPDJ2ZLM6A, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.837060829 (593h9m54.708143874s) [noise=true , address=145.79.227.26:4654   , piece_id=BW4T4E5IUD23JHUYFWK5SPHIYGVTNYONRUELBISVOBWBOIMXIBKA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.837545151 (593h9m54.708628196s) [noise=true , address=145.79.227.25:4617   , piece_id=6NLXI4HJ5JFBLOEW6VLE2JK7GOWUZT7RVHLSCTDNJSRFFGMV2NPQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.840782325 (593h9m54.71186536s) [noise=true , address=145.79.227.9:4367    , piece_id=ABA6LT3Z27LBIME27DFTMR3NY7GX7WR533ETWUVGRDFLURC4CHMQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.857888759 (593h9m54.728971804s) [noise=true , address=145.79.227.52:4446   , piece_id=WJ6ZVNDN64YEW3SYLYLVXDSX3LUFDBSFNI4DIH3L4JUEPWF7QIGA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.904941345 (593h9m54.77602439s) [noise=true , address=145.79.227.7:4293    , piece_id=NM27NO2Z3SXLC4U5JWPAERLRZIZFIALKXYNR5QFLI54BGAKR63MQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.981725374 (593h9m54.852808419s) [noise=true , address=77.90.31.11:4553     , piece_id=QKDQ2GGGXEKGQFB6H6YQ2TWZIHWIBKAUPLNKQRPZ4FKXPHXMMGQQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:55.057483561 (593h9m54.928566606s) [noise=true , address=77.90.31.14:4664     , piece_id=YF5IBB3QE3MTR6OQ5DAGSMVDYHST3CCFGGE3DI24OH654XBNBKHQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:55.057511172 (593h9m54.928594207s) [seg_pos={0 0}, upload=520005]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:38:55.057839127 (593h9m54.928922173s) [noise=true , address=45.91.116.12:4226    , piece_id=XGPB2RECAZXGPH5ZGYY65G4QKADGEJ77F6HK264FZ3DXMT5H32GA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 30 08:38:55.057848956 (593h9m54.928931991s) [noise=true , address=45.91.116.12:4226    , piece_id=XGPB2RECAZXGPH5ZGYY65G4QKADGEJ77F6HK264FZ3DXMT5H32GA, seg_pos={0 0}, upload=520005]: Long tail cancellation due to optimal threshold
Dec 30 08:38:55.058261752 (593h9m54.929344787s) [noise=true , address=45.91.116.11:4117    , piece_id=OGCLJCCITRRPMWUSWXGEKKK6Y7RXIZ4ALTPVNI7DZGQ7VYGKKKWQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:38:55.058269977 (593h9m54.929353013s) [noise=true , address=45.91.116.11:4117    , piece_id=OGCLJCCITRRPMWUSWXGEKKK6Y7RXIZ4ALTPVNI7DZGQ7VYGKKKWQ, seg_pos={0 0}, upload=520005]: Long tail cancellation due to optimal threshold
Dec 30 08:38:55.058457766 (593h9m54.929540821s) [noise=true , address=45.91.116.13:4154    , piece_id=RGZ4GXMIUNBHC7CY5ACSJNBEPA6Z7ET7BLPN46L6LSO67GCD5G4Q, seg_pos={0 0}, upload=520005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 08:38:55.058480288 (593h9m54.929563343s) [noise=true , address=45.91.116.13:4154    , piece_id=RGZ4GXMIUNBHC7CY5ACSJNBEPA6Z7ET7BLPN46L6LSO67GCD5G4Q, seg_pos={0 0}, upload=520005]: Long tail cancellation due to optimal threshold
Dec 30 08:38:55.058509634 (593h9m54.929592669s) [seg_pos={0 0}, upload=520005]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:38:55.058631006 (593h9m54.929714041s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:38:55.058639031 (593h9m54.929722076s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:38:55.358337175 (593h9m55.22942021s) [upload=520005]: Done uploading segments...
Dec 30 08:38:55.358356722 (593h9m55.229439757s) [upload=520005]: Upload finished. err: <nil>
Duration: 5.69611199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [..X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.X.X]
duration: 
30.003446031s
trace: 
https://jaeger.dev.storj.io/trace/23b49ebff208b3b6
FAIL (2025-12-30 08:39:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: QKDQ2GGGXEKGQFB6H6YQ2TWZIHWIBKAUPLNKQRPZ4FKXPHXMMGQQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: YF5IBB3QE3MTR6OQ5DAGSMVDYHST3CCFGGE3DI24OH654XBNBKHQ) context deadline exceeded
======LOGS======
Dec 30 08:38:52.544991460 (593h9m52.416074495s) [upload=520005]: Starting upload
Dec 30 08:38:52.545073707 (593h9m52.416156742s) [upload=520005]: Done starting upload
Dec 30 08:38:52.545177675 (593h9m52.41626072s) [upload=520005]: Uploading segments...
Dec 30 08:38:52.545339312 (593h9m52.416422357s) [upload=520005]: Got next segment. Inline: false
Dec 30 08:38:52.545414255 (593h9m52.41649729s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:38:52.545421569 (593h9m52.416504614s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:38:52.821369549 (593h9m52.692452584s) [seg_pos={0 0}, upload=520005]: Begin upload segment...
Dec 30 08:38:52.821415245 (593h9m52.692498281s) [seg_pos={0 0}, upload=520005]: Done begin upload segment.
Dec 30 08:38:52.821441636 (593h9m52.692524671s) [noise=true , address=145.79.227.42:4121   , piece_id=7YIOKDFQ2YO3IIPOVNZNYQI4UVLREEJTYIN5DANIBRDPDJ2ZLM6A, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.821447878 (593h9m52.692530923s) [noise=true , address=145.79.227.10:4401   , piece_id=Y3NCRNKL4RL7U56GXMBM5KFN4NMW3UZDRRS663ASPEOOKA77Z5TA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.822332231 (593h9m52.693415276s) [noise=true , address=45.91.116.13:4154    , piece_id=RGZ4GXMIUNBHC7CY5ACSJNBEPA6Z7ET7BLPN46L6LSO67GCD5G4Q, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.822775375 (593h9m52.69385842s) [noise=true , address=145.79.227.17:4970   , piece_id=VRYMFHFWHY3XF7S2V2MKFMEVAQUVYWID7QX76ZJX5X4EL5XTVBIQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.823191217 (593h9m52.694274262s) [noise=true , address=145.79.227.3:4151    , piece_id=7LRODGTBCS24MZGIIC2EO57DHLOUSAMDFGCCN2YBBIMCJDQBMQRQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.823662315 (593h9m52.69474536s) [noise=true , address=145.79.227.11:4729   , piece_id=FJ34524DVCHVF53RGXHOGZONGIA5MVMH4TCT7HSYVZAAPELCHK4Q, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.824037969 (593h9m52.695121014s) [noise=true , address=145.79.227.26:4654   , piece_id=BW4T4E5IUD23JHUYFWK5SPHIYGVTNYONRUELBISVOBWBOIMXIBKA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.824446367 (593h9m52.695529412s) [noise=true , address=77.90.31.11:4553     , piece_id=QKDQ2GGGXEKGQFB6H6YQ2TWZIHWIBKAUPLNKQRPZ4FKXPHXMMGQQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.824839165 (593h9m52.6959222s) [noise=true , address=45.91.116.12:4226    , piece_id=XGPB2RECAZXGPH5ZGYY65G4QKADGEJ77F6HK264FZ3DXMT5H32GA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.825242714 (593h9m52.696325759s) [noise=true , address=145.79.227.52:4446   , piece_id=WJ6ZVNDN64YEW3SYLYLVXDSX3LUFDBSFNI4DIH3L4JUEPWF7QIGA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.825636814 (593h9m52.696719849s) [noise=true , address=145.79.227.25:4617   , piece_id=6NLXI4HJ5JFBLOEW6VLE2JK7GOWUZT7RVHLSCTDNJSRFFGMV2NPQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.826131767 (593h9m52.697214812s) [noise=true , address=77.90.31.14:4664     , piece_id=YF5IBB3QE3MTR6OQ5DAGSMVDYHST3CCFGGE3DI24OH654XBNBKHQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.826531267 (593h9m52.697614312s) [noise=true , address=145.79.227.7:4293    , piece_id=NM27NO2Z3SXLC4U5JWPAERLRZIZFIALKXYNR5QFLI54BGAKR63MQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.826968951 (593h9m52.698051996s) [noise=true , address=145.79.227.19:4077   , piece_id=YPROJJC4PXC6DDTW6BGYXIUQSXAY74G73BOP7DNEGKPSO46YM4PA, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.827500192 (593h9m52.698583238s) [noise=true , address=45.91.116.11:4117    , piece_id=OGCLJCCITRRPMWUSWXGEKKK6Y7RXIZ4ALTPVNI7DZGQ7VYGKKKWQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:52.827983984 (593h9m52.699067029s) [noise=true , address=145.79.227.9:4367    , piece_id=ABA6LT3Z27LBIME27DFTMR3NY7GX7WR533ETWUVGRDFLURC4CHMQ, seg_pos={0 0}, upload=520005]: Uploading piece...
Dec 30 08:38:54.532314612 (593h9m54.403397647s) [upload=520005]: Next returned nil segment
Dec 30 08:38:54.532331504 (593h9m54.403414539s) [upload=520005]: Waiting for error group managing segments...
Dec 30 08:38:54.813554008 (593h9m54.684637043s) [noise=true , address=145.79.227.10:4401   , piece_id=Y3NCRNKL4RL7U56GXMBM5KFN4NMW3UZDRRS663ASPEOOKA77Z5TA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.816872887 (593h9m54.687955932s) [noise=true , address=145.79.227.19:4077   , piece_id=YPROJJC4PXC6DDTW6BGYXIUQSXAY74G73BOP7DNEGKPSO46YM4PA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.823199855 (593h9m54.6942829s) [noise=true , address=145.79.227.17:4970   , piece_id=VRYMFHFWHY3XF7S2V2MKFMEVAQUVYWID7QX76ZJX5X4EL5XTVBIQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.824758714 (593h9m54.695841759s) [noise=true , address=145.79.227.11:4729   , piece_id=FJ34524DVCHVF53RGXHOGZONGIA5MVMH4TCT7HSYVZAAPELCHK4Q, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.834126101 (593h9m54.705209136s) [noise=true , address=145.79.227.3:4151    , piece_id=7LRODGTBCS24MZGIIC2EO57DHLOUSAMDFGCCN2YBBIMCJDQBMQRQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.836052599 (593h9m54.707135644s) [noise=true , address=145.79.227.42:4121   , piece_id=7YIOKDFQ2YO3IIPOVNZNYQI4UVLREEJTYIN5DANIBRDPDJ2ZLM6A, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.837060829 (593h9m54.708143874s) [noise=true , address=145.79.227.26:4654   , piece_id=BW4T4E5IUD23JHUYFWK5SPHIYGVTNYONRUELBISVOBWBOIMXIBKA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.837545151 (593h9m54.708628196s) [noise=true , address=145.79.227.25:4617   , piece_id=6NLXI4HJ5JFBLOEW6VLE2JK7GOWUZT7RVHLSCTDNJSRFFGMV2NPQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.840782325 (593h9m54.71186536s) [noise=true , address=145.79.227.9:4367    , piece_id=ABA6LT3Z27LBIME27DFTMR3NY7GX7WR533ETWUVGRDFLURC4CHMQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.857888759 (593h9m54.728971804s) [noise=true , address=145.79.227.52:4446   , piece_id=WJ6ZVNDN64YEW3SYLYLVXDSX3LUFDBSFNI4DIH3L4JUEPWF7QIGA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.904941345 (593h9m54.77602439s) [noise=true , address=145.79.227.7:4293    , piece_id=NM27NO2Z3SXLC4U5JWPAERLRZIZFIALKXYNR5QFLI54BGAKR63MQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:54.981725374 (593h9m54.852808419s) [noise=true , address=77.90.31.11:4553     , piece_id=QKDQ2GGGXEKGQFB6H6YQ2TWZIHWIBKAUPLNKQRPZ4FKXPHXMMGQQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:55.057483561 (593h9m54.928566606s) [noise=true , address=77.90.31.14:4664     , piece_id=YF5IBB3QE3MTR6OQ5DAGSMVDYHST3CCFGGE3DI24OH654XBNBKHQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: <nil>
Dec 30 08:38:55.057511172 (593h9m54.928594207s) [seg_pos={0 0}, upload=520005]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:38:55.057839127 (593h9m54.928922173s) [noise=true , address=45.91.116.12:4226    , piece_id=XGPB2RECAZXGPH5ZGYY65G4QKADGEJ77F6HK264FZ3DXMT5H32GA, seg_pos={0 0}, upload=520005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 30 08:38:55.057848956 (593h9m54.928931991s) [noise=true , address=45.91.116.12:4226    , piece_id=XGPB2RECAZXGPH5ZGYY65G4QKADGEJ77F6HK264FZ3DXMT5H32GA, seg_pos={0 0}, upload=520005]: Long tail cancellation due to optimal threshold
Dec 30 08:38:55.058261752 (593h9m54.929344787s) [noise=true , address=45.91.116.11:4117    , piece_id=OGCLJCCITRRPMWUSWXGEKKK6Y7RXIZ4ALTPVNI7DZGQ7VYGKKKWQ, seg_pos={0 0}, upload=520005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:38:55.058269977 (593h9m54.929353013s) [noise=true , address=45.91.116.11:4117    , piece_id=OGCLJCCITRRPMWUSWXGEKKK6Y7RXIZ4ALTPVNI7DZGQ7VYGKKKWQ, seg_pos={0 0}, upload=520005]: Long tail cancellation due to optimal threshold
Dec 30 08:38:55.058457766 (593h9m54.929540821s) [noise=true , address=45.91.116.13:4154    , piece_id=RGZ4GXMIUNBHC7CY5ACSJNBEPA6Z7ET7BLPN46L6LSO67GCD5G4Q, seg_pos={0 0}, upload=520005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 08:38:55.058480288 (593h9m54.929563343s) [noise=true , address=45.91.116.13:4154    , piece_id=RGZ4GXMIUNBHC7CY5ACSJNBEPA6Z7ET7BLPN46L6LSO67GCD5G4Q, seg_pos={0 0}, upload=520005]: Long tail cancellation due to optimal threshold
Dec 30 08:38:55.058509634 (593h9m54.929592669s) [seg_pos={0 0}, upload=520005]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:38:55.058631006 (593h9m54.929714041s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:38:55.058639031 (593h9m54.929722076s) [upload=520005]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:38:55.358337175 (593h9m55.22942021s) [upload=520005]: Done uploading segments...
Dec 30 08:38:55.358356722 (593h9m55.229439757s) [upload=520005]: Upload finished. err: <nil>
Duration: 30.00347136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...X................X...............................X.......]
duration: 
40.4564105s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:39:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 40.456442309s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [XX.......X..X..X..................X.......................X.]
duration: 
3.53072725s
trace: 
https://jaeger.dev.storj.io/trace/5bb2358f8fab9298
FAIL (2025-12-30 08:39:03 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.530757186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [....X................X...............................X......]
duration: 
33.866671897s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:38:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 33.866706232s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [.XX.......X..X..X..................X.......................X]
duration: 
5.001312377s
trace: 
https://jaeger.dev.storj.io/trace/5bb2358f8fab9298
FAIL (2025-12-30 08:38:00 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001335782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [...X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.X.]
duration: 
5.894275827s
trace: 
https://jaeger.dev.storj.io/trace/52d4da75996c61f9
FAIL (2025-12-30 08:37:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: GKTM4LRQLU26YKHR72RO75G2A4LP32MFW5V6DVJTVCTWPLZELUIA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: UXTCAIYPXSD5UT64XKGI533J7JRQJZAB6G4735WLIAZQOTIBB3IA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: CFNAFSYB3COIRYLOSH4O55JWEEXCXZ7RPLIRDP77JQBQZQPYK4IQ) context deadline exceeded
======LOGS======
19980]: Starting upload
Dec 30 08:36:55.043256819 (593h7m54.914339864s) [upload=519980]: Done starting upload
Dec 30 08:36:55.043297998 (593h7m54.914381033s) [upload=519980]: Uploading segments...
Dec 30 08:36:55.043672521 (593h7m54.914755556s) [upload=519980]: Got next segment. Inline: false
Dec 30 08:36:55.043725491 (593h7m54.914808536s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:36:55.043731673 (593h7m54.914814728s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:36:55.287761547 (593h7m55.158844592s) [seg_pos={0 0}, upload=519980]: Begin upload segment...
Dec 30 08:36:55.287821490 (593h7m55.158904525s) [seg_pos={0 0}, upload=519980]: Done begin upload segment.
Dec 30 08:36:55.287876165 (593h7m55.15895921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZBQCJS6VJ2P24BNSFGVLOFV5I6FICPSOBLDL5MOMD22ZZWQZHUKA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.287885793 (593h7m55.158968848s) [noise=true , address=45.91.116.6:4974     , piece_id=GKTM4LRQLU26YKHR72RO75G2A4LP32MFW5V6DVJTVCTWPLZELUIA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.288523598 (593h7m55.159606643s) [noise=true , address=145.79.227.33:4870   , piece_id=UV65MF3G3HXPW24HHZV3MAFD7BSEMVA6JKXE5C4F2RQRVMNNC6AQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.288845461 (593h7m55.159928506s) [noise=true , address=45.91.116.10:4081    , piece_id=TAF3VUK74RY2SKRUBLKU6TKGDNZE243TH6UE3PIR2KWJIFZRP4MQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289222949 (593h7m55.160305995s) [noise=true , address=45.91.116.4:4939     , piece_id=2VTRATN4J3D67ROBQHVI6O53GVTTTDX5P3SHYMUUHMNQVP4GZ7PA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289424132 (593h7m55.160507177s) [noise=true , address=145.79.227.35:4482   , piece_id=OOE4YUDZP353QAOJASHM344BWQV24OZKL6C32R5NRS3ACDU73WKQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289583526 (593h7m55.160666561s) [noise=true , address=145.79.227.37:4566   , piece_id=CLMEVQTDUDYUQMNGSMKZQD5AYKIIP5WMTZRFBEBWG64VLT6BF2HQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289857638 (593h7m55.160940703s) [noise=true , address=145.79.227.8:4331    , piece_id=IDSK7KHCFIW6E5TXRLRHBEPKRL7SDUULX6RNIB4MZVRCTICIMEDA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289994338 (593h7m55.161077393s) [noise=true , address=145.79.227.28:4726   , piece_id=NQGMFQ7RHD33VZ42PMWBTFFLKD3ULOD5KMHB5SEJCNIUKH7RTBVQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.290377488 (593h7m55.161460533s) [noise=true , address=45.91.116.5:4298     , piece_id=NG7FV3BQEUX5FT7ICWT3VFN5BMP44KYKU6ZRYDXCPZ72U7SURPBQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.290674413 (593h7m55.161757468s) [noise=true , address=45.91.116.12:4225    , piece_id=AEZL64YAR57HK5SAKKYTX67XBNHEKKM2W6NOXX7DW7GZ6YUSTB6A, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.290736621 (593h7m55.161819666s) [noise=true , address=77.90.31.11:4554     , piece_id=UXTCAIYPXSD5UT64XKGI533J7JRQJZAB6G4735WLIAZQOTIBB3IA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.291101897 (593h7m55.162184942s) [noise=true , address=77.90.31.21:4878     , piece_id=CFNAFSYB3COIRYLOSH4O55JWEEXCXZ7RPLIRDP77JQBQZQPYK4IQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.291550582 (593h7m55.162633627s) [noise=true , address=145.79.227.32:4834   , piece_id=ZKDIVOL7M75N2ALGPDL2TT3GYTNFCL3DQAFXWQLZZ767WIMIDUMQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.291835294 (593h7m55.162918329s) [noise=true , address=145.79.227.13:4802   , piece_id=6PBUMIO452FSXYOVPZM5XAG7PZSS4HNULL5CDFETAKOPSREVYZ2A, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.292324786 (593h7m55.163407821s) [noise=true , address=77.90.31.13:4629     , piece_id=J43O7GBNDTIYUPX6L3PYVLCKIX5YQI4A4SI62XHSNIE6AWBR3XGA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:56.994448200 (593h7m56.865531245s) [upload=519980]: Next returned nil segment
Dec 30 08:36:56.994461806 (593h7m56.865544841s) [upload=519980]: Waiting for error group managing segments...
Dec 30 08:36:57.273434736 (593h7m57.144517771s) [noise=true , address=145.79.227.13:4802   , piece_id=6PBUMIO452FSXYOVPZM5XAG7PZSS4HNULL5CDFETAKOPSREVYZ2A, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.279497080 (593h7m57.150580125s) [noise=true , address=145.79.227.28:4726   , piece_id=NQGMFQ7RHD33VZ42PMWBTFFLKD3ULOD5KMHB5SEJCNIUKH7RTBVQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.294183906 (593h7m57.165266951s) [noise=true , address=145.79.227.32:4834   , piece_id=ZKDIVOL7M75N2ALGPDL2TT3GYTNFCL3DQAFXWQLZZ767WIMIDUMQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.304079159 (593h7m57.175162204s) [noise=true , address=145.79.227.37:4566   , piece_id=CLMEVQTDUDYUQMNGSMKZQD5AYKIIP5WMTZRFBEBWG64VLT6BF2HQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.308852097 (593h7m57.179935152s) [noise=true , address=145.79.227.8:4331    , piece_id=IDSK7KHCFIW6E5TXRLRHBEPKRL7SDUULX6RNIB4MZVRCTICIMEDA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.311129283 (593h7m57.182212318s) [noise=true , address=145.79.227.35:4482   , piece_id=OOE4YUDZP353QAOJASHM344BWQV24OZKL6C32R5NRS3ACDU73WKQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.449101155 (593h7m57.3201842s) [noise=true , address=77.90.31.11:4554     , piece_id=UXTCAIYPXSD5UT64XKGI533J7JRQJZAB6G4735WLIAZQOTIBB3IA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.463552323 (593h7m57.334635358s) [noise=true , address=77.90.31.21:4878     , piece_id=CFNAFSYB3COIRYLOSH4O55JWEEXCXZ7RPLIRDP77JQBQZQPYK4IQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.647195139 (593h7m57.518278184s) [noise=true , address=45.91.116.4:4939     , piece_id=2VTRATN4J3D67ROBQHVI6O53GVTTTDX5P3SHYMUUHMNQVP4GZ7PA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.656737258 (593h7m57.527820293s) [noise=true , address=77.90.31.10:4519     , piece_id=ZBQCJS6VJ2P24BNSFGVLOFV5I6FICPSOBLDL5MOMD22ZZWQZHUKA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.670638949 (593h7m57.541721984s) [noise=true , address=45.91.116.10:4081    , piece_id=TAF3VUK74RY2SKRUBLKU6TKGDNZE243TH6UE3PIR2KWJIFZRP4MQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.732330430 (593h7m57.603413475s) [noise=true , address=45.91.116.6:4974     , piece_id=GKTM4LRQLU26YKHR72RO75G2A4LP32MFW5V6DVJTVCTWPLZELUIA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.813670605 (593h7m57.68475364s) [noise=true , address=45.91.116.12:4225    , piece_id=AEZL64YAR57HK5SAKKYTX67XBNHEKKM2W6NOXX7DW7GZ6YUSTB6A, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.813691085 (593h7m57.68477412s) [seg_pos={0 0}, upload=519980]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:36:57.813930280 (593h7m57.685013326s) [noise=true , address=77.90.31.13:4629     , piece_id=J43O7GBNDTIYUPX6L3PYVLCKIX5YQI4A4SI62XHSNIE6AWBR3XGA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:41918->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:41918->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41918->77.90.31.13:4629: use of closed network connection
Dec 30 08:36:57.813936642 (593h7m57.685019677s) [noise=true , address=77.90.31.13:4629     , piece_id=J43O7GBNDTIYUPX6L3PYVLCKIX5YQI4A4SI62XHSNIE6AWBR3XGA, seg_pos={0 0}, upload=519980]: Long tail cancellation due to optimal threshold
Dec 30 08:36:57.814189133 (593h7m57.685272198s) [noise=true , address=145.79.227.33:4870   , piece_id=UV65MF3G3HXPW24HHZV3MAFD7BSEMVA6JKXE5C4F2RQRVMNNC6AQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 30 08:36:57.814208861 (593h7m57.685291906s) [noise=true , address=145.79.227.33:4870   , piece_id=UV65MF3G3HXPW24HHZV3MAFD7BSEMVA6JKXE5C4F2RQRVMNNC6AQ, seg_pos={0 0}, upload=519980]: Long tail cancellation due to optimal threshold
Dec 30 08:36:57.814288793 (593h7m57.685371828s) [noise=true , address=45.91.116.5:4298     , piece_id=NG7FV3BQEUX5FT7ICWT3VFN5BMP44KYKU6ZRYDXCPZ72U7SURPBQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:56980->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 08:36:57.814296918 (593h7m57.685379963s) [noise=true , address=45.91.116.5:4298     , piece_id=NG7FV3BQEUX5FT7ICWT3VFN5BMP44KYKU6ZRYDXCPZ72U7SURPBQ, seg_pos={0 0}, upload=519980]: Long tail cancellation due to optimal threshold
Dec 30 08:36:57.814319491 (593h7m57.685402526s) [seg_pos={0 0}, upload=519980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:36:57.814392090 (593h7m57.685475135s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:36:57.814397520 (593h7m57.685480565s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:36:58.099052790 (593h7m57.970135835s) [upload=519980]: Done uploading segments...
Dec 30 08:36:58.099079692 (593h7m57.970162727s) [upload=519980]: Upload finished. err: <nil>
Duration: 5.894307318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.....X................X...............................X.....]
duration: 
40.789758889s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:37:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 40.78977521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [....X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.X]
duration: 
30.001977793s
trace: 
https://jaeger.dev.storj.io/trace/52d4da75996c61f9
FAIL (2025-12-30 08:37:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: GKTM4LRQLU26YKHR72RO75G2A4LP32MFW5V6DVJTVCTWPLZELUIA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: UXTCAIYPXSD5UT64XKGI533J7JRQJZAB6G4735WLIAZQOTIBB3IA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: CFNAFSYB3COIRYLOSH4O55JWEEXCXZ7RPLIRDP77JQBQZQPYK4IQ) context deadline exceeded
======LOGS======
19980]: Starting upload
Dec 30 08:36:55.043256819 (593h7m54.914339864s) [upload=519980]: Done starting upload
Dec 30 08:36:55.043297998 (593h7m54.914381033s) [upload=519980]: Uploading segments...
Dec 30 08:36:55.043672521 (593h7m54.914755556s) [upload=519980]: Got next segment. Inline: false
Dec 30 08:36:55.043725491 (593h7m54.914808536s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:36:55.043731673 (593h7m54.914814728s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:36:55.287761547 (593h7m55.158844592s) [seg_pos={0 0}, upload=519980]: Begin upload segment...
Dec 30 08:36:55.287821490 (593h7m55.158904525s) [seg_pos={0 0}, upload=519980]: Done begin upload segment.
Dec 30 08:36:55.287876165 (593h7m55.15895921s) [noise=true , address=77.90.31.10:4519     , piece_id=ZBQCJS6VJ2P24BNSFGVLOFV5I6FICPSOBLDL5MOMD22ZZWQZHUKA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.287885793 (593h7m55.158968848s) [noise=true , address=45.91.116.6:4974     , piece_id=GKTM4LRQLU26YKHR72RO75G2A4LP32MFW5V6DVJTVCTWPLZELUIA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.288523598 (593h7m55.159606643s) [noise=true , address=145.79.227.33:4870   , piece_id=UV65MF3G3HXPW24HHZV3MAFD7BSEMVA6JKXE5C4F2RQRVMNNC6AQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.288845461 (593h7m55.159928506s) [noise=true , address=45.91.116.10:4081    , piece_id=TAF3VUK74RY2SKRUBLKU6TKGDNZE243TH6UE3PIR2KWJIFZRP4MQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289222949 (593h7m55.160305995s) [noise=true , address=45.91.116.4:4939     , piece_id=2VTRATN4J3D67ROBQHVI6O53GVTTTDX5P3SHYMUUHMNQVP4GZ7PA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289424132 (593h7m55.160507177s) [noise=true , address=145.79.227.35:4482   , piece_id=OOE4YUDZP353QAOJASHM344BWQV24OZKL6C32R5NRS3ACDU73WKQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289583526 (593h7m55.160666561s) [noise=true , address=145.79.227.37:4566   , piece_id=CLMEVQTDUDYUQMNGSMKZQD5AYKIIP5WMTZRFBEBWG64VLT6BF2HQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289857638 (593h7m55.160940703s) [noise=true , address=145.79.227.8:4331    , piece_id=IDSK7KHCFIW6E5TXRLRHBEPKRL7SDUULX6RNIB4MZVRCTICIMEDA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.289994338 (593h7m55.161077393s) [noise=true , address=145.79.227.28:4726   , piece_id=NQGMFQ7RHD33VZ42PMWBTFFLKD3ULOD5KMHB5SEJCNIUKH7RTBVQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.290377488 (593h7m55.161460533s) [noise=true , address=45.91.116.5:4298     , piece_id=NG7FV3BQEUX5FT7ICWT3VFN5BMP44KYKU6ZRYDXCPZ72U7SURPBQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.290674413 (593h7m55.161757468s) [noise=true , address=45.91.116.12:4225    , piece_id=AEZL64YAR57HK5SAKKYTX67XBNHEKKM2W6NOXX7DW7GZ6YUSTB6A, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.290736621 (593h7m55.161819666s) [noise=true , address=77.90.31.11:4554     , piece_id=UXTCAIYPXSD5UT64XKGI533J7JRQJZAB6G4735WLIAZQOTIBB3IA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.291101897 (593h7m55.162184942s) [noise=true , address=77.90.31.21:4878     , piece_id=CFNAFSYB3COIRYLOSH4O55JWEEXCXZ7RPLIRDP77JQBQZQPYK4IQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.291550582 (593h7m55.162633627s) [noise=true , address=145.79.227.32:4834   , piece_id=ZKDIVOL7M75N2ALGPDL2TT3GYTNFCL3DQAFXWQLZZ767WIMIDUMQ, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.291835294 (593h7m55.162918329s) [noise=true , address=145.79.227.13:4802   , piece_id=6PBUMIO452FSXYOVPZM5XAG7PZSS4HNULL5CDFETAKOPSREVYZ2A, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:55.292324786 (593h7m55.163407821s) [noise=true , address=77.90.31.13:4629     , piece_id=J43O7GBNDTIYUPX6L3PYVLCKIX5YQI4A4SI62XHSNIE6AWBR3XGA, seg_pos={0 0}, upload=519980]: Uploading piece...
Dec 30 08:36:56.994448200 (593h7m56.865531245s) [upload=519980]: Next returned nil segment
Dec 30 08:36:56.994461806 (593h7m56.865544841s) [upload=519980]: Waiting for error group managing segments...
Dec 30 08:36:57.273434736 (593h7m57.144517771s) [noise=true , address=145.79.227.13:4802   , piece_id=6PBUMIO452FSXYOVPZM5XAG7PZSS4HNULL5CDFETAKOPSREVYZ2A, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.279497080 (593h7m57.150580125s) [noise=true , address=145.79.227.28:4726   , piece_id=NQGMFQ7RHD33VZ42PMWBTFFLKD3ULOD5KMHB5SEJCNIUKH7RTBVQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.294183906 (593h7m57.165266951s) [noise=true , address=145.79.227.32:4834   , piece_id=ZKDIVOL7M75N2ALGPDL2TT3GYTNFCL3DQAFXWQLZZ767WIMIDUMQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.304079159 (593h7m57.175162204s) [noise=true , address=145.79.227.37:4566   , piece_id=CLMEVQTDUDYUQMNGSMKZQD5AYKIIP5WMTZRFBEBWG64VLT6BF2HQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.308852097 (593h7m57.179935152s) [noise=true , address=145.79.227.8:4331    , piece_id=IDSK7KHCFIW6E5TXRLRHBEPKRL7SDUULX6RNIB4MZVRCTICIMEDA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.311129283 (593h7m57.182212318s) [noise=true , address=145.79.227.35:4482   , piece_id=OOE4YUDZP353QAOJASHM344BWQV24OZKL6C32R5NRS3ACDU73WKQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.449101155 (593h7m57.3201842s) [noise=true , address=77.90.31.11:4554     , piece_id=UXTCAIYPXSD5UT64XKGI533J7JRQJZAB6G4735WLIAZQOTIBB3IA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.463552323 (593h7m57.334635358s) [noise=true , address=77.90.31.21:4878     , piece_id=CFNAFSYB3COIRYLOSH4O55JWEEXCXZ7RPLIRDP77JQBQZQPYK4IQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.647195139 (593h7m57.518278184s) [noise=true , address=45.91.116.4:4939     , piece_id=2VTRATN4J3D67ROBQHVI6O53GVTTTDX5P3SHYMUUHMNQVP4GZ7PA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.656737258 (593h7m57.527820293s) [noise=true , address=77.90.31.10:4519     , piece_id=ZBQCJS6VJ2P24BNSFGVLOFV5I6FICPSOBLDL5MOMD22ZZWQZHUKA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.670638949 (593h7m57.541721984s) [noise=true , address=45.91.116.10:4081    , piece_id=TAF3VUK74RY2SKRUBLKU6TKGDNZE243TH6UE3PIR2KWJIFZRP4MQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.732330430 (593h7m57.603413475s) [noise=true , address=45.91.116.6:4974     , piece_id=GKTM4LRQLU26YKHR72RO75G2A4LP32MFW5V6DVJTVCTWPLZELUIA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.813670605 (593h7m57.68475364s) [noise=true , address=45.91.116.12:4225    , piece_id=AEZL64YAR57HK5SAKKYTX67XBNHEKKM2W6NOXX7DW7GZ6YUSTB6A, seg_pos={0 0}, upload=519980]: Done uploading piece. err: <nil>
Dec 30 08:36:57.813691085 (593h7m57.68477412s) [seg_pos={0 0}, upload=519980]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:36:57.813930280 (593h7m57.685013326s) [noise=true , address=77.90.31.13:4629     , piece_id=J43O7GBNDTIYUPX6L3PYVLCKIX5YQI4A4SI62XHSNIE6AWBR3XGA, seg_pos={0 0}, upload=519980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:41918->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:41918->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41918->77.90.31.13:4629: use of closed network connection
Dec 30 08:36:57.813936642 (593h7m57.685019677s) [noise=true , address=77.90.31.13:4629     , piece_id=J43O7GBNDTIYUPX6L3PYVLCKIX5YQI4A4SI62XHSNIE6AWBR3XGA, seg_pos={0 0}, upload=519980]: Long tail cancellation due to optimal threshold
Dec 30 08:36:57.814189133 (593h7m57.685272198s) [noise=true , address=145.79.227.33:4870   , piece_id=UV65MF3G3HXPW24HHZV3MAFD7BSEMVA6JKXE5C4F2RQRVMNNC6AQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 30 08:36:57.814208861 (593h7m57.685291906s) [noise=true , address=145.79.227.33:4870   , piece_id=UV65MF3G3HXPW24HHZV3MAFD7BSEMVA6JKXE5C4F2RQRVMNNC6AQ, seg_pos={0 0}, upload=519980]: Long tail cancellation due to optimal threshold
Dec 30 08:36:57.814288793 (593h7m57.685371828s) [noise=true , address=45.91.116.5:4298     , piece_id=NG7FV3BQEUX5FT7ICWT3VFN5BMP44KYKU6ZRYDXCPZ72U7SURPBQ, seg_pos={0 0}, upload=519980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:56980->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 08:36:57.814296918 (593h7m57.685379963s) [noise=true , address=45.91.116.5:4298     , piece_id=NG7FV3BQEUX5FT7ICWT3VFN5BMP44KYKU6ZRYDXCPZ72U7SURPBQ, seg_pos={0 0}, upload=519980]: Long tail cancellation due to optimal threshold
Dec 30 08:36:57.814319491 (593h7m57.685402526s) [seg_pos={0 0}, upload=519980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:36:57.814392090 (593h7m57.685475135s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:36:57.814397520 (593h7m57.685480565s) [upload=519980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:36:58.099052790 (593h7m57.970135835s) [upload=519980]: Done uploading segments...
Dec 30 08:36:58.099079692 (593h7m57.970162727s) [upload=519980]: Upload finished. err: <nil>
Duration: 30.002005215s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [X.XX.......X..X..X..................X.......................]
duration: 
5.000559324s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:36:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000585944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.....X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X.]
duration: 
6.020564451s
trace: 
https://jaeger.dev.storj.io/trace/221661995209cb7f
FAIL (2025-12-30 08:36:39 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 30 08:35:36.419783108 (593h6m36.290866143s) [upload=519960]: Starting upload
Dec 30 08:35:36.419838033 (593h6m36.290921068s) [upload=519960]: Done starting upload
Dec 30 08:35:36.419955156 (593h6m36.291038191s) [upload=519960]: Uploading segments...
Dec 30 08:35:36.420082138 (593h6m36.291165173s) [upload=519960]: Got next segment. Inline: false
Dec 30 08:35:36.420116844 (593h6m36.291199879s) [upload=519960]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:35:36.420121483 (593h6m36.291204528s) [upload=519960]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:35:36.669782028 (593h6m36.540865073s) [seg_pos={0 0}, upload=519960]: Begin upload segment...
Dec 30 08:35:36.669860137 (593h6m36.540943172s) [seg_pos={0 0}, upload=519960]: Done begin upload segment.
Dec 30 08:35:36.669888821 (593h6m36.540971856s) [noise=true , address=145.79.227.20:4438   , piece_id=WBLSCY6FVKANQ5ZFNYPAOHUCK5KBSKPH2UZABU4J5B7EBKO3YXYQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.669899512 (593h6m36.540982557s) [noise=true , address=145.79.227.15:4874   , piece_id=XVXPLR7Y2NKXUKXY5JUB626WOYTRZKXTQOVN6BNIY7HWSWL2A3SA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.669967331 (593h6m36.541050376s) [noise=true , address=145.79.227.48:4303   , piece_id=7AE5JRIOR32XDCNBHBWVFP47ENZUXCTZWXRGCDC7F5YUSJE7DFBA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.670551232 (593h6m36.541634277s) [noise=true , address=145.79.227.23:4546   , piece_id=PN3YG6DGV2QNAWBG4BVZQNSUVRVVD4PMRWDE2XOZIXI7H4VUR5YQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.670995549 (593h6m36.542078584s) [noise=true , address=145.79.227.21:4475   , piece_id=P6N22JF2JOJI6KTMRVBZ6YJ6ZZHOKEO56ZI5YWHRU2HAN5J777WA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.671102342 (593h6m36.542185387s) [noise=true , address=77.90.31.11:4554     , piece_id=QTWYBEMU3IOKG2SHGETW5SYE4J2TNWYLEEJTM7UXLVBHBJ4KF43Q, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.671756057 (593h6m36.542839092s) [noise=true , address=45.91.116.10:4082    , piece_id=5NCXMUZA27ME3VW36ZNMSRFVXCFRLJCZPX33MNMF4E4HNNR62JSA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.671811984 (593h6m36.542895029s) [noise=true , address=145.79.227.47:4267   , piece_id=6WYCMEOIRKOGDV6UPOO347Q5LBBRJM7Q257MNVW2OMQEI5EBW7YQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.672398570 (593h6m36.543481605s) [noise=true , address=45.91.116.6:4976     , piece_id=7DCJPTWX7ULRNKLU2MFLTM77MFAGFL6X3SXS7VDLI63SPZQACIHA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.672449167 (593h6m36.543532212s) [noise=true , address=77.90.31.16:4736     , piece_id=GJFNFW3UKC6RW3AV2IHM2CWKFK4H4XEESCC7FCSNIZBI3AUFKSQA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.673178716 (593h6m36.544261751s) [noise=true , address=145.79.227.13:4804   , piece_id=4VSGHANN46OGH5J5V42V42BTDBERWXTPD4LWYSH374NG43VB4IPQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.673838913 (593h6m36.544921948s) [noise=true , address=77.90.31.10:4517     , piece_id=B3P6SQPJYJRPJEDZSKUXHTUI2ZWWVX3ZSWCJVDIJDNGF4OLBOXRA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674288730 (593h6m36.545371765s) [noise=true , address=145.79.227.17:4972   , piece_id=F7RJXY3O2OHZD3NROOOB4WAAZIU27NLAUHBS22W4MCRQJOOHT7QA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674631613 (593h6m36.545714658s) [noise=true , address=145.79.227.43:4159   , piece_id=Z2BYXB4A4CPSUEQ2X7NVX7PHW76DS7ONW4KSQMV2TYGZSZZBU2PQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674778613 (593h6m36.545861658s) [noise=true , address=145.79.227.36:4519   , piece_id=5IKMLWZRRJAXORMB57BVM7DGAFWRV5AGWQ4GNXUPOTS6XELKQXCQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674998511 (593h6m36.546081546s) [noise=true , address=145.79.227.11:4731   , piece_id=W5U5GQNNKG3HVH3KRTNLHEIAHSHL4TYGVGGLQUKP4AHEGOZXAG3A, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:38.394292391 (593h6m38.265375436s) [upload=519960]: Next returned nil segment
Dec 30 08:35:38.394309733 (593h6m38.265392768s) [upload=519960]: Waiting for error group managing segments...
Dec 30 08:35:38.675874560 (593h6m38.546957595s) [noise=true , address=145.79.227.15:4874   , piece_id=XVXPLR7Y2NKXUKXY5JUB626WOYTRZKXTQOVN6BNIY7HWSWL2A3SA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.680352497 (593h6m38.551435542s) [noise=true , address=145.79.227.48:4303   , piece_id=7AE5JRIOR32XDCNBHBWVFP47ENZUXCTZWXRGCDC7F5YUSJE7DFBA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.682156621 (593h6m38.553239667s) [noise=true , address=145.79.227.11:4731   , piece_id=W5U5GQNNKG3HVH3KRTNLHEIAHSHL4TYGVGGLQUKP4AHEGOZXAG3A, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.684389915 (593h6m38.55547296s) [noise=true , address=145.79.227.13:4804   , piece_id=4VSGHANN46OGH5J5V42V42BTDBERWXTPD4LWYSH374NG43VB4IPQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.688978902 (593h6m38.560061937s) [noise=true , address=145.79.227.47:4267   , piece_id=6WYCMEOIRKOGDV6UPOO347Q5LBBRJM7Q257MNVW2OMQEI5EBW7YQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.925557788 (593h6m38.796640833s) [noise=true , address=77.90.31.11:4554     , piece_id=QTWYBEMU3IOKG2SHGETW5SYE4J2TNWYLEEJTM7UXLVBHBJ4KF43Q, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.971547399 (593h6m38.842630464s) [noise=true , address=45.91.116.6:4976     , piece_id=7DCJPTWX7ULRNKLU2MFLTM77MFAGFL6X3SXS7VDLI63SPZQACIHA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.977899315 (593h6m38.84898235s) [noise=true , address=77.90.31.10:4517     , piece_id=B3P6SQPJYJRPJEDZSKUXHTUI2ZWWVX3ZSWCJVDIJDNGF4OLBOXRA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:39.140313950 (593h6m39.011396995s) [noise=true , address=45.91.116.10:4082    , piece_id=5NCXMUZA27ME3VW36ZNMSRFVXCFRLJCZPX33MNMF4E4HNNR62JSA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:39.585513704 (593h6m39.456596739s) [noise=true , address=77.90.31.16:4736     , piece_id=GJFNFW3UKC6RW3AV2IHM2CWKFK4H4XEESCC7FCSNIZBI3AUFKSQA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:40.506834912 (593h6m40.377917947s) [noise=true , address=145.79.227.43:4159   , piece_id=Z2BYXB4A4CPSUEQ2X7NVX7PHW76DS7ONW4KSQMV2TYGZSZZBU2PQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:53.119785274 (593h6m52.990868319s) [noise=true , address=145.79.227.21:4475   , piece_id=P6N22JF2JOJI6KTMRVBZ6YJ6ZZHOKEO56ZI5YWHRU2HAN5J777WA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:36:05.730409718 (593h7m5.601492763s) [noise=true , address=145.79.227.23:4546   , piece_id=PN3YG6DGV2QNAWBG4BVZQNSUVRVVD4PMRWDE2XOZIXI7H4VUR5YQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:57280->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:57280->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57280->145.79.227.23:4546: use of closed network connection
Dec 30 08:36:05.730573741 (593h7m5.601656786s) [noise=true , address=145.79.227.20:4438   , piece_id=WBLSCY6FVKANQ5ZFNYPAOHUCK5KBSKPH2UZABU4J5B7EBKO3YXYQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:50536->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:50536->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50536->145.79.227.20:4438: use of closed network connection
Dec 30 08:36:05.730614158 (593h7m5.601697203s) [noise=true , address=145.79.227.36:4519   , piece_id=5IKMLWZRRJAXORMB57BVM7DGAFWRV5AGWQ4GNXUPOTS6XELKQXCQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; context deadline exceeded; EOF
Dec 30 08:36:05.730768402 (593h7m5.601851457s) [noise=true , address=145.79.227.17:4972   , piece_id=F7RJXY3O2OHZD3NROOOB4WAAZIU27NLAUHBS22W4MCRQJOOHT7QA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:49126->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:49126->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49126->145.79.227.17:4972: use of closed network connection
Dec 30 08:36:05.730801094 (593h7m5.601884139s) [seg_pos={0 0}, upload=519960]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 30 08:36:05.974131235 (593h7m5.84521427s) [upload=519960]: Done uploading segments...
Dec 30 08:36:05.974171662 (593h7m5.845254707s) [upload=519960]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.02061188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [......X................X...............................X....]
duration: 
40.23223419s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:36:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 40.232251111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [......X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...X]
duration: 
30.246025473s
trace: 
https://jaeger.dev.storj.io/trace/221661995209cb7f
FAIL (2025-12-30 08:36:05 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 30 08:35:36.419783108 (593h6m36.290866143s) [upload=519960]: Starting upload
Dec 30 08:35:36.419838033 (593h6m36.290921068s) [upload=519960]: Done starting upload
Dec 30 08:35:36.419955156 (593h6m36.291038191s) [upload=519960]: Uploading segments...
Dec 30 08:35:36.420082138 (593h6m36.291165173s) [upload=519960]: Got next segment. Inline: false
Dec 30 08:35:36.420116844 (593h6m36.291199879s) [upload=519960]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:35:36.420121483 (593h6m36.291204528s) [upload=519960]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:35:36.669782028 (593h6m36.540865073s) [seg_pos={0 0}, upload=519960]: Begin upload segment...
Dec 30 08:35:36.669860137 (593h6m36.540943172s) [seg_pos={0 0}, upload=519960]: Done begin upload segment.
Dec 30 08:35:36.669888821 (593h6m36.540971856s) [noise=true , address=145.79.227.20:4438   , piece_id=WBLSCY6FVKANQ5ZFNYPAOHUCK5KBSKPH2UZABU4J5B7EBKO3YXYQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.669899512 (593h6m36.540982557s) [noise=true , address=145.79.227.15:4874   , piece_id=XVXPLR7Y2NKXUKXY5JUB626WOYTRZKXTQOVN6BNIY7HWSWL2A3SA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.669967331 (593h6m36.541050376s) [noise=true , address=145.79.227.48:4303   , piece_id=7AE5JRIOR32XDCNBHBWVFP47ENZUXCTZWXRGCDC7F5YUSJE7DFBA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.670551232 (593h6m36.541634277s) [noise=true , address=145.79.227.23:4546   , piece_id=PN3YG6DGV2QNAWBG4BVZQNSUVRVVD4PMRWDE2XOZIXI7H4VUR5YQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.670995549 (593h6m36.542078584s) [noise=true , address=145.79.227.21:4475   , piece_id=P6N22JF2JOJI6KTMRVBZ6YJ6ZZHOKEO56ZI5YWHRU2HAN5J777WA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.671102342 (593h6m36.542185387s) [noise=true , address=77.90.31.11:4554     , piece_id=QTWYBEMU3IOKG2SHGETW5SYE4J2TNWYLEEJTM7UXLVBHBJ4KF43Q, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.671756057 (593h6m36.542839092s) [noise=true , address=45.91.116.10:4082    , piece_id=5NCXMUZA27ME3VW36ZNMSRFVXCFRLJCZPX33MNMF4E4HNNR62JSA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.671811984 (593h6m36.542895029s) [noise=true , address=145.79.227.47:4267   , piece_id=6WYCMEOIRKOGDV6UPOO347Q5LBBRJM7Q257MNVW2OMQEI5EBW7YQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.672398570 (593h6m36.543481605s) [noise=true , address=45.91.116.6:4976     , piece_id=7DCJPTWX7ULRNKLU2MFLTM77MFAGFL6X3SXS7VDLI63SPZQACIHA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.672449167 (593h6m36.543532212s) [noise=true , address=77.90.31.16:4736     , piece_id=GJFNFW3UKC6RW3AV2IHM2CWKFK4H4XEESCC7FCSNIZBI3AUFKSQA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.673178716 (593h6m36.544261751s) [noise=true , address=145.79.227.13:4804   , piece_id=4VSGHANN46OGH5J5V42V42BTDBERWXTPD4LWYSH374NG43VB4IPQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.673838913 (593h6m36.544921948s) [noise=true , address=77.90.31.10:4517     , piece_id=B3P6SQPJYJRPJEDZSKUXHTUI2ZWWVX3ZSWCJVDIJDNGF4OLBOXRA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674288730 (593h6m36.545371765s) [noise=true , address=145.79.227.17:4972   , piece_id=F7RJXY3O2OHZD3NROOOB4WAAZIU27NLAUHBS22W4MCRQJOOHT7QA, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674631613 (593h6m36.545714658s) [noise=true , address=145.79.227.43:4159   , piece_id=Z2BYXB4A4CPSUEQ2X7NVX7PHW76DS7ONW4KSQMV2TYGZSZZBU2PQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674778613 (593h6m36.545861658s) [noise=true , address=145.79.227.36:4519   , piece_id=5IKMLWZRRJAXORMB57BVM7DGAFWRV5AGWQ4GNXUPOTS6XELKQXCQ, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:36.674998511 (593h6m36.546081546s) [noise=true , address=145.79.227.11:4731   , piece_id=W5U5GQNNKG3HVH3KRTNLHEIAHSHL4TYGVGGLQUKP4AHEGOZXAG3A, seg_pos={0 0}, upload=519960]: Uploading piece...
Dec 30 08:35:38.394292391 (593h6m38.265375436s) [upload=519960]: Next returned nil segment
Dec 30 08:35:38.394309733 (593h6m38.265392768s) [upload=519960]: Waiting for error group managing segments...
Dec 30 08:35:38.675874560 (593h6m38.546957595s) [noise=true , address=145.79.227.15:4874   , piece_id=XVXPLR7Y2NKXUKXY5JUB626WOYTRZKXTQOVN6BNIY7HWSWL2A3SA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.680352497 (593h6m38.551435542s) [noise=true , address=145.79.227.48:4303   , piece_id=7AE5JRIOR32XDCNBHBWVFP47ENZUXCTZWXRGCDC7F5YUSJE7DFBA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.682156621 (593h6m38.553239667s) [noise=true , address=145.79.227.11:4731   , piece_id=W5U5GQNNKG3HVH3KRTNLHEIAHSHL4TYGVGGLQUKP4AHEGOZXAG3A, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.684389915 (593h6m38.55547296s) [noise=true , address=145.79.227.13:4804   , piece_id=4VSGHANN46OGH5J5V42V42BTDBERWXTPD4LWYSH374NG43VB4IPQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.688978902 (593h6m38.560061937s) [noise=true , address=145.79.227.47:4267   , piece_id=6WYCMEOIRKOGDV6UPOO347Q5LBBRJM7Q257MNVW2OMQEI5EBW7YQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.925557788 (593h6m38.796640833s) [noise=true , address=77.90.31.11:4554     , piece_id=QTWYBEMU3IOKG2SHGETW5SYE4J2TNWYLEEJTM7UXLVBHBJ4KF43Q, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.971547399 (593h6m38.842630464s) [noise=true , address=45.91.116.6:4976     , piece_id=7DCJPTWX7ULRNKLU2MFLTM77MFAGFL6X3SXS7VDLI63SPZQACIHA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:38.977899315 (593h6m38.84898235s) [noise=true , address=77.90.31.10:4517     , piece_id=B3P6SQPJYJRPJEDZSKUXHTUI2ZWWVX3ZSWCJVDIJDNGF4OLBOXRA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:39.140313950 (593h6m39.011396995s) [noise=true , address=45.91.116.10:4082    , piece_id=5NCXMUZA27ME3VW36ZNMSRFVXCFRLJCZPX33MNMF4E4HNNR62JSA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:39.585513704 (593h6m39.456596739s) [noise=true , address=77.90.31.16:4736     , piece_id=GJFNFW3UKC6RW3AV2IHM2CWKFK4H4XEESCC7FCSNIZBI3AUFKSQA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:40.506834912 (593h6m40.377917947s) [noise=true , address=145.79.227.43:4159   , piece_id=Z2BYXB4A4CPSUEQ2X7NVX7PHW76DS7ONW4KSQMV2TYGZSZZBU2PQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:35:53.119785274 (593h6m52.990868319s) [noise=true , address=145.79.227.21:4475   , piece_id=P6N22JF2JOJI6KTMRVBZ6YJ6ZZHOKEO56ZI5YWHRU2HAN5J777WA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: <nil>
Dec 30 08:36:05.730409718 (593h7m5.601492763s) [noise=true , address=145.79.227.23:4546   , piece_id=PN3YG6DGV2QNAWBG4BVZQNSUVRVVD4PMRWDE2XOZIXI7H4VUR5YQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:57280->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:57280->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57280->145.79.227.23:4546: use of closed network connection
Dec 30 08:36:05.730573741 (593h7m5.601656786s) [noise=true , address=145.79.227.20:4438   , piece_id=WBLSCY6FVKANQ5ZFNYPAOHUCK5KBSKPH2UZABU4J5B7EBKO3YXYQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:50536->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:50536->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50536->145.79.227.20:4438: use of closed network connection
Dec 30 08:36:05.730614158 (593h7m5.601697203s) [noise=true , address=145.79.227.36:4519   , piece_id=5IKMLWZRRJAXORMB57BVM7DGAFWRV5AGWQ4GNXUPOTS6XELKQXCQ, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; context deadline exceeded; EOF
Dec 30 08:36:05.730768402 (593h7m5.601851457s) [noise=true , address=145.79.227.17:4972   , piece_id=F7RJXY3O2OHZD3NROOOB4WAAZIU27NLAUHBS22W4MCRQJOOHT7QA, seg_pos={0 0}, upload=519960]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:49126->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:49126->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49126->145.79.227.17:4972: use of closed network connection
Dec 30 08:36:05.730801094 (593h7m5.601884139s) [seg_pos={0 0}, upload=519960]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 30 08:36:05.974131235 (593h7m5.84521427s) [upload=519960]: Done uploading segments...
Dec 30 08:36:05.974171662 (593h7m5.845254707s) [upload=519960]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.246053497s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [XX.XX.......X..X..X..................X......................]
duration: 
3.486011533s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:35:54 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.486039466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.......X................X...............................X...]
duration: 
28.442430166s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:35:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 28.442460615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [XXX.XX.......X..X..X..................X.....................]
duration: 
3.684423912s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:35:30 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.6844497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [........X................X...............................X..]
duration: 
43.028622286s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:35:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 43.028660569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.......X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX...]
duration: 
24.996786699s
trace: 
https://jaeger.dev.storj.io/trace/36136729ad76ec5b
FAIL (2025-12-30 08:34:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA) context deadline exceeded
======LOGS======
Dec 30 08:32:08.534384588 (593h3m8.405467633s) [upload=519912]: Starting upload
Dec 30 08:32:08.534447698 (593h3m8.405530743s) [upload=519912]: Done starting upload
Dec 30 08:32:08.534521318 (593h3m8.405604353s) [upload=519912]: Uploading segments...
Dec 30 08:32:08.534648711 (593h3m8.405731756s) [upload=519912]: Got next segment. Inline: false
Dec 30 08:32:08.534678057 (593h3m8.405761092s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:32:08.534682516 (593h3m8.405765551s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:32:08.781837831 (593h3m8.652920876s) [seg_pos={0 0}, upload=519912]: Begin upload segment...
Dec 30 08:32:08.781897324 (593h3m8.652980359s) [seg_pos={0 0}, upload=519912]: Done begin upload segment.
Dec 30 08:32:08.781936688 (593h3m8.653019734s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786777406 (593h3m8.657860451s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786992746 (593h3m8.658075801s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788571413 (593h3m8.659654458s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788781202 (593h3m8.659864247s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790163714 (593h3m8.661246759s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790458135 (593h3m8.66154118s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.791712113 (593h3m8.662795148s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.793541486 (593h3m8.664624531s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795540923 (593h3m8.666623959s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795863998 (593h3m8.666947043s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797099300 (593h3m8.668182335s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797600996 (593h3m8.668684041s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.799338034 (593h3m8.670421079s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.800865973 (593h3m8.671949018s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.802212236 (593h3m8.673295281s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:10.491861718 (593h3m10.362944753s) [upload=519912]: Next returned nil segment
Dec 30 08:32:10.491887027 (593h3m10.362970072s) [upload=519912]: Waiting for error group managing segments...
Dec 30 08:32:10.773226594 (593h3m10.644309639s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.785967996 (593h3m10.657051031s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.790230743 (593h3m10.661313778s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.793903947 (593h3m10.664986992s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.798913025 (593h3m10.66999606s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.804383803 (593h3m10.675466838s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.805982457 (593h3m10.677065502s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.812244019 (593h3m10.683327084s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.864780988 (593h3m10.735864033s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.872196879 (593h3m10.743279924s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.040873749 (593h3m10.911956784s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.060517924 (593h3m10.931600959s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113543193 (593h3m10.984626248s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113579442 (593h3m10.984662477s) [seg_pos={0 0}, upload=519912]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:32:11.113952021 (593h3m10.985035056s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.113962602 (593h3m10.985045647s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.113970176 (593h3m10.985053211s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114000233 (593h3m10.985083268s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114258254 (593h3m10.985341289s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114265788 (593h3m10.985348834s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114282711 (593h3m10.985365756s) [seg_pos={0 0}, upload=519912]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:32:11.114386479 (593h3m10.985469514s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:11.114394524 (593h3m10.985477569s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:11.397895134 (593h3m11.268978179s) [upload=519912]: Done uploading segments...
Dec 30 08:32:11.397918729 (593h3m11.269001774s) [upload=519912]: Upload finished. err: <nil>
Duration: 24.996814202s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
10 out of 60 is failed (threshold=6) [XXXX.XX.......X..X..X..................X....................]
duration: 
5.001432136s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:34:22 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001456833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [........X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX..]
duration: 
6.039460293s
trace: 
https://jaeger.dev.storj.io/trace/36136729ad76ec5b
FAIL (2025-12-30 08:33:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA) context deadline exceeded
======LOGS======
Dec 30 08:32:08.534384588 (593h3m8.405467633s) [upload=519912]: Starting upload
Dec 30 08:32:08.534447698 (593h3m8.405530743s) [upload=519912]: Done starting upload
Dec 30 08:32:08.534521318 (593h3m8.405604353s) [upload=519912]: Uploading segments...
Dec 30 08:32:08.534648711 (593h3m8.405731756s) [upload=519912]: Got next segment. Inline: false
Dec 30 08:32:08.534678057 (593h3m8.405761092s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:32:08.534682516 (593h3m8.405765551s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:32:08.781837831 (593h3m8.652920876s) [seg_pos={0 0}, upload=519912]: Begin upload segment...
Dec 30 08:32:08.781897324 (593h3m8.652980359s) [seg_pos={0 0}, upload=519912]: Done begin upload segment.
Dec 30 08:32:08.781936688 (593h3m8.653019734s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786777406 (593h3m8.657860451s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786992746 (593h3m8.658075801s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788571413 (593h3m8.659654458s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788781202 (593h3m8.659864247s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790163714 (593h3m8.661246759s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790458135 (593h3m8.66154118s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.791712113 (593h3m8.662795148s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.793541486 (593h3m8.664624531s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795540923 (593h3m8.666623959s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795863998 (593h3m8.666947043s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797099300 (593h3m8.668182335s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797600996 (593h3m8.668684041s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.799338034 (593h3m8.670421079s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.800865973 (593h3m8.671949018s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.802212236 (593h3m8.673295281s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:10.491861718 (593h3m10.362944753s) [upload=519912]: Next returned nil segment
Dec 30 08:32:10.491887027 (593h3m10.362970072s) [upload=519912]: Waiting for error group managing segments...
Dec 30 08:32:10.773226594 (593h3m10.644309639s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.785967996 (593h3m10.657051031s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.790230743 (593h3m10.661313778s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.793903947 (593h3m10.664986992s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.798913025 (593h3m10.66999606s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.804383803 (593h3m10.675466838s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.805982457 (593h3m10.677065502s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.812244019 (593h3m10.683327084s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.864780988 (593h3m10.735864033s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.872196879 (593h3m10.743279924s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.040873749 (593h3m10.911956784s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.060517924 (593h3m10.931600959s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113543193 (593h3m10.984626248s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113579442 (593h3m10.984662477s) [seg_pos={0 0}, upload=519912]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:32:11.113952021 (593h3m10.985035056s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.113962602 (593h3m10.985045647s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.113970176 (593h3m10.985053211s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114000233 (593h3m10.985083268s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114258254 (593h3m10.985341289s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114265788 (593h3m10.985348834s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114282711 (593h3m10.985365756s) [seg_pos={0 0}, upload=519912]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:32:11.114386479 (593h3m10.985469514s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:11.114394524 (593h3m10.985477569s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:11.397895134 (593h3m11.268978179s) [upload=519912]: Done uploading segments...
Dec 30 08:32:11.397918729 (593h3m11.269001774s) [upload=519912]: Upload finished. err: <nil>
Duration: 6.03948978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.........X................X...............................X.]
duration: 
15.326221551s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:33:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 15.326238633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
11 out of 60 is failed (threshold=6) [XXXXX.XX.......X..X..X..................X...................]
duration: 
3.514612323s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:33:26 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.514637421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..........X................X...............................X]
duration: 
1m0.003384335s
trace: 
https://jaeger.dev.storj.io/trace/1113591c3748aab1
FAIL (2025-12-30 08:33:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:32:18.539435911 (593h3m18.410518956s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.539973164 (593h3m18.411056209s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542032806 (593h3m18.413115851s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.542560010 (593h3m18.413643045s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543038551 (593h3m18.414121596s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.543662148 (593h3m18.414745193s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.544611786 (593h3m18.415694831s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.545255863 (593h3m18.416338908s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.546147992 (593h3m18.417231037s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.547596380 (593h3m18.418679415s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552332197 (593h3m18.423415243s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.552833252 (593h3m18.423916307s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.557983159 (593h3m18.429066194s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.560395091 (593h3m18.431478136s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.566556542 (593h3m18.437639578s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:18.568845291 (593h3m18.439928336s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Uploading piece...
Dec 30 08:32:19.943059343 (593h3m19.814142388s) [upload=519914]: Next returned nil segment
Dec 30 08:32:19.943076195 (593h3m19.81415924s) [upload=519914]: Waiting for error group managing segments...
Dec 30 08:32:20.317478529 (593h3m20.188561564s) [noise=true , address=64.49.13.27:4963     , piece_id=EMD4MPICXSEWCHM7SSQR4N2YLV66M322PVBPYDFJLPBCBDWGBPNA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.440545900 (593h3m20.311628945s) [noise=true , address=145.79.227.29:4970   , piece_id=GBP6YCQFCBK2OXST7IXF7JNFJRDCE6L4Z3TWMOBUG73IXSD52H3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.457618128 (593h3m20.328701173s) [noise=true , address=145.79.227.5:4229    , piece_id=OWKMJDB4TXKIBXWKNOY2XVVYFHPL66XI5D6WLQCXTSABR3K2MJ3Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.501151192 (593h3m20.372234227s) [noise=true , address=145.79.227.43:4167   , piece_id=OWZPR4LZW76NN7RBMSX4AV2LXW6GT4D4CX3A5KFEW7JDYJ7FWT7Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.518633972 (593h3m20.389717017s) [noise=true , address=145.79.227.19:4085   , piece_id=PGNRGW7WBSXZXEVKB3PF6AJN2EMKZU6FJLKFSQBPNXR2OFBHK4ZA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.538821953 (593h3m20.409904989s) [noise=true , address=145.79.227.35:4494   , piece_id=KWB7H7YX5I527TOQT5ZRBDF77CH7XDEKL33QRCXV7QJEDBQDSNSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.628921688 (593h3m20.500004723s) [noise=true , address=145.79.227.40:4080   , piece_id=L6MLTHKXKDQYWKJ4NT6TZPIHZYGWHF4VJ74QMLUEPU6RFUSAF4LQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.716397186 (593h3m20.587480231s) [noise=true , address=77.90.31.11:4588     , piece_id=FYFR3GHDH5RFYTQYXKQ5HTPNLSTQIZIVKCGBUCC4P65LMGGUJTEA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.717901300 (593h3m20.588984335s) [noise=true , address=77.90.31.17:4803     , piece_id=XOQYTNQUHUJZQQSOSBWTJVTQNPFFEXJ6DVKNWELDXKNBYCVX4B6Q, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:20.844119180 (593h3m20.715202225s) [noise=true , address=77.90.31.18:4818     , piece_id=LEBQ2KW4NSQRT24UTXXOIDYJT5ZTHLNUSRYQDXXCV2SYK3FB3SPQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.119837773 (593h3m20.990920818s) [noise=true , address=145.79.227.33:4872   , piece_id=WZ7Y4KAIOY6DSOQF7M6RRKO5E62DIIZRA3H7BIB5AMYQKKACYMOA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.296191751 (593h3m21.167274786s) [noise=true , address=145.79.227.30:4775   , piece_id=E4SE3NKQCNLSJ74J42WM5R6FHK2HPK3FXONQT6O4QHIOWNLZ3YLQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.360424160 (593h3m21.231507195s) [noise=true , address=193.149.164.143:4697 , piece_id=FH46UG5MDHAIAK75LGDQTJOM5W5522AZ6PSPVTTB7KIO2JQACRQQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374519980 (593h3m21.245603025s) [noise=true , address=145.79.227.49:4370   , piece_id=LNVLZH2TZ4XCBVYYPXIMR5IYGBZP3BIPBEGFEA2KQM47H54FPNIQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: <nil>
Dec 30 08:32:21.374541301 (593h3m21.245624346s) [seg_pos={0 0}, upload=519914]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:32:21.374729459 (593h3m21.245812494s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374735951 (593h3m21.245818996s) [noise=true , address=77.90.31.15:4704     , piece_id=YRR3D24CENYOXWXHX24EFB6NZESDBYLDYYPLCH55J3CGWW5MCS5A, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374846582 (593h3m21.245929617s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374851732 (593h3m21.245934777s) [noise=true , address=145.79.227.44:4203   , piece_id=AKCVH33PM7RSX6ASHKIYTZTV5737UUWKVNRSZSMSFCQDOKZND6XA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.374985096 (593h3m21.246068131s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:21.374992690 (593h3m21.246075736s) [noise=true , address=77.90.31.14:4683     , piece_id=VPPD3QCX3UHPHO3Y3GL7E7G4UQU253PWZTYXFNLX2GRBVCZR22AQ, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375059287 (593h3m21.246142332s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->64.49.13.169:4706: use of closed network connection
Dec 30 08:32:21.375081780 (593h3m21.246164845s) [noise=true , address=64.49.13.169:4706    , piece_id=X2DDHNX5VLL7Y4ABVL2IPSSBG33QG7AKSHKADHB4OTMAXZUXTMSA, seg_pos={0 0}, upload=519914]: Long tail cancellation due to optimal threshold
Dec 30 08:32:21.375172352 (593h3m21.246255387s) [seg_pos={0 0}, upload=519914]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:32:21.375453638 (593h3m21.246536683s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:21.375463036 (593h3m21.246546091s) [upload=519914]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:21.677401727 (593h3m21.548484762s) [upload=519914]: Done uploading segments...
Dec 30 08:32:21.677421695 (593h3m21.54850474s) [upload=519914]: Upload finished. err: <nil>
Duration: 1m0.003422247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.........X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX.]
duration: 
5.748105913s
trace: 
https://jaeger.dev.storj.io/trace/36136729ad76ec5b
FAIL (2025-12-30 08:32:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA) context deadline exceeded
======LOGS======
Dec 30 08:32:08.534384588 (593h3m8.405467633s) [upload=519912]: Starting upload
Dec 30 08:32:08.534447698 (593h3m8.405530743s) [upload=519912]: Done starting upload
Dec 30 08:32:08.534521318 (593h3m8.405604353s) [upload=519912]: Uploading segments...
Dec 30 08:32:08.534648711 (593h3m8.405731756s) [upload=519912]: Got next segment. Inline: false
Dec 30 08:32:08.534678057 (593h3m8.405761092s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:32:08.534682516 (593h3m8.405765551s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:32:08.781837831 (593h3m8.652920876s) [seg_pos={0 0}, upload=519912]: Begin upload segment...
Dec 30 08:32:08.781897324 (593h3m8.652980359s) [seg_pos={0 0}, upload=519912]: Done begin upload segment.
Dec 30 08:32:08.781936688 (593h3m8.653019734s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786777406 (593h3m8.657860451s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786992746 (593h3m8.658075801s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788571413 (593h3m8.659654458s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788781202 (593h3m8.659864247s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790163714 (593h3m8.661246759s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790458135 (593h3m8.66154118s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.791712113 (593h3m8.662795148s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.793541486 (593h3m8.664624531s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795540923 (593h3m8.666623959s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795863998 (593h3m8.666947043s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797099300 (593h3m8.668182335s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797600996 (593h3m8.668684041s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.799338034 (593h3m8.670421079s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.800865973 (593h3m8.671949018s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.802212236 (593h3m8.673295281s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:10.491861718 (593h3m10.362944753s) [upload=519912]: Next returned nil segment
Dec 30 08:32:10.491887027 (593h3m10.362970072s) [upload=519912]: Waiting for error group managing segments...
Dec 30 08:32:10.773226594 (593h3m10.644309639s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.785967996 (593h3m10.657051031s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.790230743 (593h3m10.661313778s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.793903947 (593h3m10.664986992s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.798913025 (593h3m10.66999606s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.804383803 (593h3m10.675466838s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.805982457 (593h3m10.677065502s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.812244019 (593h3m10.683327084s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.864780988 (593h3m10.735864033s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.872196879 (593h3m10.743279924s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.040873749 (593h3m10.911956784s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.060517924 (593h3m10.931600959s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113543193 (593h3m10.984626248s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113579442 (593h3m10.984662477s) [seg_pos={0 0}, upload=519912]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:32:11.113952021 (593h3m10.985035056s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.113962602 (593h3m10.985045647s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.113970176 (593h3m10.985053211s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114000233 (593h3m10.985083268s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114258254 (593h3m10.985341289s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114265788 (593h3m10.985348834s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114282711 (593h3m10.985365756s) [seg_pos={0 0}, upload=519912]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:32:11.114386479 (593h3m10.985469514s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:11.114394524 (593h3m10.985477569s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:11.397895134 (593h3m11.268978179s) [upload=519912]: Done uploading segments...
Dec 30 08:32:11.397918729 (593h3m11.269001774s) [upload=519912]: Upload finished. err: <nil>
Duration: 5.748137204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..........X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XXX]
duration: 
30.002143266s
trace: 
https://jaeger.dev.storj.io/trace/36136729ad76ec5b
FAIL (2025-12-30 08:32:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA) context deadline exceeded
======LOGS======
Dec 30 08:32:08.534384588 (593h3m8.405467633s) [upload=519912]: Starting upload
Dec 30 08:32:08.534447698 (593h3m8.405530743s) [upload=519912]: Done starting upload
Dec 30 08:32:08.534521318 (593h3m8.405604353s) [upload=519912]: Uploading segments...
Dec 30 08:32:08.534648711 (593h3m8.405731756s) [upload=519912]: Got next segment. Inline: false
Dec 30 08:32:08.534678057 (593h3m8.405761092s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:32:08.534682516 (593h3m8.405765551s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:32:08.781837831 (593h3m8.652920876s) [seg_pos={0 0}, upload=519912]: Begin upload segment...
Dec 30 08:32:08.781897324 (593h3m8.652980359s) [seg_pos={0 0}, upload=519912]: Done begin upload segment.
Dec 30 08:32:08.781936688 (593h3m8.653019734s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786777406 (593h3m8.657860451s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.786992746 (593h3m8.658075801s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788571413 (593h3m8.659654458s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.788781202 (593h3m8.659864247s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790163714 (593h3m8.661246759s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.790458135 (593h3m8.66154118s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.791712113 (593h3m8.662795148s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.793541486 (593h3m8.664624531s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795540923 (593h3m8.666623959s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.795863998 (593h3m8.666947043s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797099300 (593h3m8.668182335s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.797600996 (593h3m8.668684041s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.799338034 (593h3m8.670421079s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.800865973 (593h3m8.671949018s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:08.802212236 (593h3m8.673295281s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Uploading piece...
Dec 30 08:32:10.491861718 (593h3m10.362944753s) [upload=519912]: Next returned nil segment
Dec 30 08:32:10.491887027 (593h3m10.362970072s) [upload=519912]: Waiting for error group managing segments...
Dec 30 08:32:10.773226594 (593h3m10.644309639s) [noise=true , address=145.79.227.10:4401   , piece_id=ZCBOA4TKC7HH4NC2SZRTV6QVLN4OL3YQ7KILMQOPQZWAUA465QSQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.785967996 (593h3m10.657051031s) [noise=true , address=145.79.227.17:4972   , piece_id=CTTLZBCVULTJ7OICVXYDPVQXGKU64DWMOW5CPZB3TQCFJQI2AS6Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.790230743 (593h3m10.661313778s) [noise=true , address=145.79.227.11:4731   , piece_id=YSRCKAGM4ARA6V32D4XMSXZ5S2NS5QCD26AFXNUJQO2ZL2AWGNLQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.793903947 (593h3m10.664986992s) [noise=true , address=145.79.227.13:4804   , piece_id=5WF7B7ISMN3CHUGUFOFYGAYAPCSLCDUTUD554POSW43AVQRZCH5Q, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.798913025 (593h3m10.66999606s) [noise=true , address=145.79.227.8:4329    , piece_id=ZRKOKYTW64TIDOEXQ2VON2HPFQQGFHHFZ4FO67A4IM4ED7XT5IZA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.804383803 (593h3m10.675466838s) [noise=true , address=145.79.227.5:4221    , piece_id=QASKIMRTZBL64IOL3TNAX3OCC4R7OZ3UHHIHQUK4P4KYX4OBZ2UQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.805982457 (593h3m10.677065502s) [noise=true , address=145.79.227.18:4006   , piece_id=ZUDSSGQUZITS3MHALPU6ERFQJDFXFQPA2COT6LD7MTYWPAZCZJZQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.812244019 (593h3m10.683327084s) [noise=true , address=145.79.227.3:4151    , piece_id=RXEGEGUYLSW32ZNDUPWHCOJDWAMPDKKMKPYJVJ4WLL3ISN454MQQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.864780988 (593h3m10.735864033s) [noise=true , address=145.79.227.15:4876   , piece_id=OADVGCYRQ7AQAFSWWDDSZZCO5KMELKWDCAGK4SHTSKJHP2PA47EA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:10.872196879 (593h3m10.743279924s) [noise=true , address=145.79.227.35:4483   , piece_id=P277VPOZFNATDWYCTPUODSMFL4LWRLDQE5KGQYEZK2KG2NYKCQAA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.040873749 (593h3m10.911956784s) [noise=true , address=77.90.31.19:4841     , piece_id=7QXBV3QLKGH4NUKZBMFMUYW3DISSR2NKTA5XFA5YWXMQPQSSOCQA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.060517924 (593h3m10.931600959s) [noise=true , address=77.90.31.16:4736     , piece_id=72PTFVUD4M6CIKPALK5RZMN6NFIDTHKOKSGKYF7WIQZNIFV2HDRA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113543193 (593h3m10.984626248s) [noise=true , address=45.91.116.8:4046     , piece_id=S7QZ6BLGH62UN5QH2E7XYXLADUCZEIKHLO6ALLOMU5OQ72C3TNFA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: <nil>
Dec 30 08:32:11.113579442 (593h3m10.984662477s) [seg_pos={0 0}, upload=519912]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:32:11.113952021 (593h3m10.985035056s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.113962602 (593h3m10.985045647s) [noise=true , address=45.91.116.10:4081    , piece_id=5PPO2FBMWP6CYHG3GPZUWNNODWBZYOFJVEQDNW3MF2SW6LXW7LMQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.113970176 (593h3m10.985053211s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114000233 (593h3m10.985083268s) [noise=true , address=45.91.116.6:4973     , piece_id=VP2SML7IQARHIJSSNA5X52RXJZEMVK4SSIMUY4RXZOURJVPTPDCA, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114258254 (593h3m10.985341289s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 30 08:32:11.114265788 (593h3m10.985348834s) [noise=true , address=77.90.31.12:4589     , piece_id=TYA7Z7XOG3KP3KP34X7PHJIUMCVWEIU7GAAUSF6Z3PEIWWQIE7LQ, seg_pos={0 0}, upload=519912]: Long tail cancellation due to optimal threshold
Dec 30 08:32:11.114282711 (593h3m10.985365756s) [seg_pos={0 0}, upload=519912]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:32:11.114386479 (593h3m10.985469514s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:32:11.114394524 (593h3m10.985477569s) [upload=519912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:32:11.397895134 (593h3m11.268978179s) [upload=519912]: Done uploading segments...
Dec 30 08:32:11.397918729 (593h3m11.269001774s) [upload=519912]: Upload finished. err: <nil>
Duration: 30.00215584s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
12 out of 60 is failed (threshold=6) [XXXXXX.XX.......X..X..X..................X..................]
duration: 
3.61773792s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:32:14 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.617968068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [...........X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.XX]
duration: 
30.003399908s
trace: 
https://jaeger.dev.storj.io/trace/56236ff04fe360fe
FAIL (2025-12-30 08:32:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 2WNUAFKAWIYC4RLB2PR5ADFGNB63M4GYHC7S4SHF53N56ZVNJXQA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EWGE3DKBMXPQN4TWOCJMI2DM2E65GETYKSZ7PKT4COKNSLC7OSYQ) context deadline exceeded
======LOGS======
Dec 30 08:31:38.306029231 (593h2m38.177112267s) [upload=519903]: Starting upload
Dec 30 08:31:38.306164779 (593h2m38.177247834s) [upload=519903]: Done starting upload
Dec 30 08:31:38.306552037 (593h2m38.177635082s) [upload=519903]: Uploading segments...
Dec 30 08:31:38.307709320 (593h2m38.178792385s) [upload=519903]: Got next segment. Inline: false
Dec 30 08:31:38.307820462 (593h2m38.178903507s) [upload=519903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:31:38.307837043 (593h2m38.178920098s) [upload=519903]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:31:38.557917750 (593h2m38.429000795s) [seg_pos={0 0}, upload=519903]: Begin upload segment...
Dec 30 08:31:38.557971612 (593h2m38.429054657s) [seg_pos={0 0}, upload=519903]: Done begin upload segment.
Dec 30 08:31:38.558077714 (593h2m38.429160749s) [noise=true , address=45.91.116.13:4156    , piece_id=6XJUOCIBQOPLU56B5ZU3ECHZCWJTGDB5DFKI5FNHLR2C4E37DL5A, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.559117253 (593h2m38.430200298s) [noise=true , address=145.79.227.15:4873   , piece_id=F35TB4RSYO3JAWJR4AJ7HGF3CUEIPC3I3J6YA63E4DMI2OSO4AAA, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.559234526 (593h2m38.430317571s) [noise=true , address=145.79.227.11:4730   , piece_id=O44MRO6NAFNCXTWUYF37AR7JGQGNV7GEESSXND2XO7LUC4COY6CQ, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.559967662 (593h2m38.431050707s) [noise=true , address=45.91.116.4:4940     , piece_id=WLX2NU6ZRLRIAIHIGEMC2ZIO3KMQ5762YKT5T33QFW67CIVWO64Q, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.560422468 (593h2m38.431505503s) [noise=true , address=77.90.31.14:4661     , piece_id=4QTR7ZOE7HKYJHOVMCTNTE7CHEARP7CXCWSWYIBBKCHZTYUNXCKA, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.560869900 (593h2m38.431952945s) [noise=true , address=145.79.227.23:4547   , piece_id=C7ZOJD5ZEPAYQZCTZMBYPBIM2AUUNLCUTLN2BO2PIAIQQ2KFJCZA, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.561338964 (593h2m38.432422009s) [noise=true , address=45.91.116.6:4974     , piece_id=EWGE3DKBMXPQN4TWOCJMI2DM2E65GETYKSZ7PKT4COKNSLC7OSYQ, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.561869224 (593h2m38.432952269s) [noise=true , address=145.79.227.50:4374   , piece_id=A2X3PDRTVTTWTTA5PXDLWADF2SV2W7QSO7ABMK7MJZOJAW556WTA, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.561951219 (593h2m38.433034264s) [noise=true , address=145.79.227.20:4437   , piece_id=FNQ6SPEJ3KW7TU6AYPZRGMH6776JFSUOSJAQARH5QMNLXGZPSBEA, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.563499478 (593h2m38.434582523s) [noise=true , address=145.79.227.32:4834   , piece_id=JCGUEQA475WCLGZ6HKR2TDRLG6FLWHAWCR6EEZD6J65ADNKIIF6Q, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.564509852 (593h2m38.435592887s) [noise=true , address=145.79.227.48:4303   , piece_id=Q3U6AEOPV3XQKM6K6ZVAJ6KTJUX7QJM4OG7X2XRX5RCUZHZETM4Q, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.565903015 (593h2m38.43698606s) [noise=true , address=145.79.227.46:4602   , piece_id=NDFZTFEGTQOM4F4M3KFN5QFQPT2RVMJ5GSJKVHWLSUZN2JDZSM5Q, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.566636171 (593h2m38.437719216s) [noise=true , address=145.79.227.16:4910   , piece_id=533NGA7DJH6EBWZHTNIOOVLTKI4APS7D5JU4CB7MU3DECCOTQPRQ, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.567515144 (593h2m38.438598189s) [noise=true , address=77.90.31.19:4841     , piece_id=IEJV4HD3SQRSDRHX5CVMO3HCOCGFAJCREDQMYC3KHZOCCU24ZMYQ, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.568680704 (593h2m38.439763749s) [noise=true , address=45.91.116.11:4119    , piece_id=2WNUAFKAWIYC4RLB2PR5ADFGNB63M4GYHC7S4SHF53N56ZVNJXQA, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:38.569412297 (593h2m38.440495342s) [noise=true , address=77.90.31.10:4518     , piece_id=HMY54L5P4IO7SO36GZUZOMEV2DPEK53OQFE7QUPOPEMNVRNELBTA, seg_pos={0 0}, upload=519903]: Uploading piece...
Dec 30 08:31:40.272487739 (593h2m40.143570774s) [upload=519903]: Next returned nil segment
Dec 30 08:31:40.272502998 (593h2m40.143586043s) [upload=519903]: Waiting for error group managing segments...
Dec 30 08:31:40.559565634 (593h2m40.430648669s) [noise=true , address=145.79.227.50:4374   , piece_id=A2X3PDRTVTTWTTA5PXDLWADF2SV2W7QSO7ABMK7MJZOJAW556WTA, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.561126055 (593h2m40.432209101s) [noise=true , address=145.79.227.46:4602   , piece_id=NDFZTFEGTQOM4F4M3KFN5QFQPT2RVMJ5GSJKVHWLSUZN2JDZSM5Q, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.563479446 (593h2m40.434562481s) [noise=true , address=145.79.227.48:4303   , piece_id=Q3U6AEOPV3XQKM6K6ZVAJ6KTJUX7QJM4OG7X2XRX5RCUZHZETM4Q, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.583069017 (593h2m40.454152062s) [noise=true , address=145.79.227.23:4547   , piece_id=C7ZOJD5ZEPAYQZCTZMBYPBIM2AUUNLCUTLN2BO2PIAIQQ2KFJCZA, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.638559592 (593h2m40.509642637s) [noise=true , address=145.79.227.15:4873   , piece_id=F35TB4RSYO3JAWJR4AJ7HGF3CUEIPC3I3J6YA63E4DMI2OSO4AAA, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.638738112 (593h2m40.509821157s) [noise=true , address=145.79.227.32:4834   , piece_id=JCGUEQA475WCLGZ6HKR2TDRLG6FLWHAWCR6EEZD6J65ADNKIIF6Q, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.741684077 (593h2m40.612767122s) [noise=true , address=77.90.31.10:4518     , piece_id=HMY54L5P4IO7SO36GZUZOMEV2DPEK53OQFE7QUPOPEMNVRNELBTA, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.829360849 (593h2m40.700443894s) [noise=true , address=77.90.31.19:4841     , piece_id=IEJV4HD3SQRSDRHX5CVMO3HCOCGFAJCREDQMYC3KHZOCCU24ZMYQ, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.907734516 (593h2m40.778817561s) [noise=true , address=45.91.116.6:4974     , piece_id=EWGE3DKBMXPQN4TWOCJMI2DM2E65GETYKSZ7PKT4COKNSLC7OSYQ, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.950784450 (593h2m40.821867485s) [noise=true , address=45.91.116.4:4940     , piece_id=WLX2NU6ZRLRIAIHIGEMC2ZIO3KMQ5762YKT5T33QFW67CIVWO64Q, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:40.989359964 (593h2m40.860443009s) [noise=true , address=45.91.116.11:4119    , piece_id=2WNUAFKAWIYC4RLB2PR5ADFGNB63M4GYHC7S4SHF53N56ZVNJXQA, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:41.055849512 (593h2m40.926932557s) [noise=true , address=45.91.116.13:4156    , piece_id=6XJUOCIBQOPLU56B5ZU3ECHZCWJTGDB5DFKI5FNHLR2C4E37DL5A, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:41.556784256 (593h2m41.427867291s) [noise=true , address=77.90.31.14:4661     , piece_id=4QTR7ZOE7HKYJHOVMCTNTE7CHEARP7CXCWSWYIBBKCHZTYUNXCKA, seg_pos={0 0}, upload=519903]: Done uploading piece. err: <nil>
Dec 30 08:31:41.556809073 (593h2m41.427892138s) [seg_pos={0 0}, upload=519903]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:31:41.557025445 (593h2m41.42810849s) [noise=true , address=145.79.227.20:4437   , piece_id=FNQ6SPEJ3KW7TU6AYPZRGMH6776JFSUOSJAQARH5QMNLXGZPSBEA, seg_pos={0 0}, upload=519903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF
Dec 30 08:31:41.557033380 (593h2m41.428116415s) [noise=true , address=145.79.227.20:4437   , piece_id=FNQ6SPEJ3KW7TU6AYPZRGMH6776JFSUOSJAQARH5QMNLXGZPSBEA, seg_pos={0 0}, upload=519903]: Long tail cancellation due to optimal threshold
Dec 30 08:31:41.557081973 (593h2m41.428165028s) [noise=true , address=145.79.227.11:4730   , piece_id=O44MRO6NAFNCXTWUYF37AR7JGQGNV7GEESSXND2XO7LUC4COY6CQ, seg_pos={0 0}, upload=519903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF
Dec 30 08:31:41.557092623 (593h2m41.428175678s) [noise=true , address=145.79.227.11:4730   , piece_id=O44MRO6NAFNCXTWUYF37AR7JGQGNV7GEESSXND2XO7LUC4COY6CQ, seg_pos={0 0}, upload=519903]: Long tail cancellation due to optimal threshold
Dec 30 08:31:41.557358459 (593h2m41.428441495s) [noise=true , address=145.79.227.16:4910   , piece_id=533NGA7DJH6EBWZHTNIOOVLTKI4APS7D5JU4CB7MU3DECCOTQPRQ, seg_pos={0 0}, upload=519903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF
Dec 30 08:31:41.557367126 (593h2m41.428450161s) [noise=true , address=145.79.227.16:4910   , piece_id=533NGA7DJH6EBWZHTNIOOVLTKI4APS7D5JU4CB7MU3DECCOTQPRQ, seg_pos={0 0}, upload=519903]: Long tail cancellation due to optimal threshold
Dec 30 08:31:41.557391633 (593h2m41.428474678s) [seg_pos={0 0}, upload=519903]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:31:41.557567908 (593h2m41.428650953s) [upload=519903]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:31:41.557574410 (593h2m41.428657466s) [upload=519903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:31:41.842254510 (593h2m41.713337565s) [upload=519903]: Done uploading segments...
Dec 30 08:31:41.842279747 (593h2m41.713362792s) [upload=519903]: Upload finished. err: <nil>
Duration: 30.003417891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...........X................X...............................]
duration: 
15.304366976s
trace: 
https://jaeger.dev.storj.io/trace/3368aa43178d2f5a
FAIL (2025-12-30 08:32:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment
Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments...
Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982   , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289     , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766   , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection
Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection
Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection
Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments...
Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil>
Duration: 15.304392515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
12 out of 60 is failed (threshold=6) [.XXXXXX.XX.......X..X..X..................X.................]
duration: 
3.453101776s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:31:48 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.453130983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............X................X..............................]
duration: 
20.197310058s
trace: 
https://jaeger.dev.storj.io/trace/3368aa43178d2f5a
FAIL (2025-12-30 08:31:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment
Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments...
Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982   , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289     , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766   , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection
Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection
Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection
Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments...
Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil>
Duration: 20.197329575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X................X.............................]
duration: 
40.733715816s
trace: 
https://jaeger.dev.storj.io/trace/3368aa43178d2f5a
FAIL (2025-12-30 08:31:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment
Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments...
Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982   , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289     , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766   , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection
Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection
Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection
Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments...
Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil>
Duration: 40.733740905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [............X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.X]
duration: 
30.003822564s
trace: 
https://jaeger.dev.storj.io/trace/472965e6099cda8b
FAIL (2025-12-30 08:31:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4CBWZOSVXYTKGWZLD43WZHVSAOW3BQMHDAMVU6EBJR3VQP57E3NQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: QF6GO3AQNQTSBFXGKDS6VPVZIP3AXAV2352MCDHIYGMZKLCIXQSA) context deadline exceeded
======LOGS======
19885]: Starting upload
Dec 30 08:30:37.039919715 (593h1m36.91100276s) [upload=519885]: Done starting upload
Dec 30 08:30:37.039946976 (593h1m36.911030011s) [upload=519885]: Uploading segments...
Dec 30 08:30:37.040153209 (593h1m36.911236254s) [upload=519885]: Got next segment. Inline: false
Dec 30 08:30:37.040189949 (593h1m36.911272994s) [upload=519885]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:30:37.040196291 (593h1m36.911279336s) [upload=519885]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:30:37.284974530 (593h1m37.156057565s) [seg_pos={0 0}, upload=519885]: Begin upload segment...
Dec 30 08:30:37.285028874 (593h1m37.156111919s) [seg_pos={0 0}, upload=519885]: Done begin upload segment.
Dec 30 08:30:37.285054863 (593h1m37.156137898s) [noise=true , address=77.90.31.21:4880     , piece_id=4CBWZOSVXYTKGWZLD43WZHVSAOW3BQMHDAMVU6EBJR3VQP57E3NQ, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.285073519 (593h1m37.156156564s) [noise=true , address=77.90.31.10:4520     , piece_id=OFYIT72YDB2KRWAEXW7UDYGJSK3YJISG4PCDZR6XX2TA4ETHDYDQ, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.285486455 (593h1m37.15656949s) [noise=true , address=145.79.227.17:4970   , piece_id=U6G7WVHRYWDQUOJW5E2EN5M7TR7ZRX6LUSPMVT74Q7J6ZSHTWHSA, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.285757101 (593h1m37.156840136s) [noise=true , address=145.79.227.36:4519   , piece_id=SMG5RZQQKQG2CDL7WSNANW3DQN5AIUPPBF6NFMS4DIDNEZ2IBPSQ, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.286070508 (593h1m37.157153553s) [noise=true , address=145.79.227.46:4603   , piece_id=CTWJBLR6IH5RG7BV5MQTXJX6DWBJIGFMKOBSL47DETGI6VED73NA, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.286350280 (593h1m37.157433325s) [noise=true , address=77.90.31.15:4697     , piece_id=654JXMB62N2NJNF6DNKAG3QMNM7JHE3HWZ3IEVPFEK3PH6YTLLGQ, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.286661663 (593h1m37.157744698s) [noise=true , address=45.91.116.12:4227    , piece_id=J7FQHX5P2AVKDJYKX7ZFIKNJAGAQIDZJZS7QSDAELKX262YZRMKQ, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.286989086 (593h1m37.158072131s) [noise=true , address=145.79.227.15:4875   , piece_id=PANTIPZJIT7SLZYARHMVW756ENDWCXBHDPF7CWZ67HIB5TMG6G2A, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.287576625 (593h1m37.15865966s) [noise=true , address=45.91.116.13:4153    , piece_id=YJN4ENFTTQQ66FR7XAFWDB3VCQ2GEW3PF7BSQLWWWBM6GWX5OT3A, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.288696558 (593h1m37.159779603s) [noise=true , address=77.90.31.16:4736     , piece_id=QF6GO3AQNQTSBFXGKDS6VPVZIP3AXAV2352MCDHIYGMZKLCIXQSA, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.288719181 (593h1m37.159802216s) [noise=true , address=145.79.227.20:4439   , piece_id=G7WRYEVLDA5EQSUAMHT3HZ2XIXNU7MFY6JLODERSP42IJBLMW7CA, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.289763229 (593h1m37.160846264s) [noise=true , address=77.90.31.19:4841     , piece_id=SVO4MSXS5TBEUNX4QQM6LNQRKPEW4THRFOAQMPTWKVAN5LBZWWVA, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.289872848 (593h1m37.160955903s) [noise=true , address=145.79.227.6:4258    , piece_id=JKZ5NBDZPOXAG5UOA2WZS6PQWMK7SY3UN5PMLS67K6BZTJAYIIUQ, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.291021705 (593h1m37.16210476s) [noise=true , address=145.79.227.10:4402   , piece_id=OZSBQRGQTUKUEIGOEOCO55UVR5OKIYYI32VYVKFM43XASMUBW62A, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.291241203 (593h1m37.162324248s) [noise=true , address=145.79.227.19:4077   , piece_id=2IQ7QWYQM3SC5NFMWM7JEGAGF2HYTXBMMSTJAD7PIUYNGCT6VE6Q, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:37.292713646 (593h1m37.163796691s) [noise=true , address=145.79.227.52:4447   , piece_id=JAZBO5V63HSN25XXZOQBR5KSFTXM3TF4AK7RWUAHXMNZBUIOFYNA, seg_pos={0 0}, upload=519885]: Uploading piece...
Dec 30 08:30:39.004754590 (593h1m38.875837646s) [upload=519885]: Next returned nil segment
Dec 30 08:30:39.004768276 (593h1m38.875851331s) [upload=519885]: Waiting for error group managing segments...
Dec 30 08:30:39.291636262 (593h1m39.162719317s) [noise=true , address=145.79.227.10:4402   , piece_id=OZSBQRGQTUKUEIGOEOCO55UVR5OKIYYI32VYVKFM43XASMUBW62A, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.291978463 (593h1m39.163061518s) [noise=true , address=145.79.227.19:4077   , piece_id=2IQ7QWYQM3SC5NFMWM7JEGAGF2HYTXBMMSTJAD7PIUYNGCT6VE6Q, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.306443587 (593h1m39.177526622s) [noise=true , address=145.79.227.6:4258    , piece_id=JKZ5NBDZPOXAG5UOA2WZS6PQWMK7SY3UN5PMLS67K6BZTJAYIIUQ, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.311949140 (593h1m39.183032185s) [noise=true , address=145.79.227.20:4439   , piece_id=G7WRYEVLDA5EQSUAMHT3HZ2XIXNU7MFY6JLODERSP42IJBLMW7CA, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.315854557 (593h1m39.186937592s) [noise=true , address=145.79.227.15:4875   , piece_id=PANTIPZJIT7SLZYARHMVW756ENDWCXBHDPF7CWZ67HIB5TMG6G2A, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.359432176 (593h1m39.230515211s) [noise=true , address=145.79.227.46:4603   , piece_id=CTWJBLR6IH5RG7BV5MQTXJX6DWBJIGFMKOBSL47DETGI6VED73NA, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.360758071 (593h1m39.231841106s) [noise=true , address=145.79.227.36:4519   , piece_id=SMG5RZQQKQG2CDL7WSNANW3DQN5AIUPPBF6NFMS4DIDNEZ2IBPSQ, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.372371906 (593h1m39.243454941s) [noise=true , address=145.79.227.52:4447   , piece_id=JAZBO5V63HSN25XXZOQBR5KSFTXM3TF4AK7RWUAHXMNZBUIOFYNA, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.459639167 (593h1m39.330722202s) [noise=true , address=77.90.31.16:4736     , piece_id=QF6GO3AQNQTSBFXGKDS6VPVZIP3AXAV2352MCDHIYGMZKLCIXQSA, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.483174783 (593h1m39.354257818s) [noise=true , address=77.90.31.21:4880     , piece_id=4CBWZOSVXYTKGWZLD43WZHVSAOW3BQMHDAMVU6EBJR3VQP57E3NQ, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.536176116 (593h1m39.407259161s) [noise=true , address=145.79.227.17:4970   , piece_id=U6G7WVHRYWDQUOJW5E2EN5M7TR7ZRX6LUSPMVT74Q7J6ZSHTWHSA, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.584356010 (593h1m39.455439045s) [noise=true , address=77.90.31.10:4520     , piece_id=OFYIT72YDB2KRWAEXW7UDYGJSK3YJISG4PCDZR6XX2TA4ETHDYDQ, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.604914917 (593h1m39.475997952s) [noise=true , address=45.91.116.13:4153    , piece_id=YJN4ENFTTQQ66FR7XAFWDB3VCQ2GEW3PF7BSQLWWWBM6GWX5OT3A, seg_pos={0 0}, upload=519885]: Done uploading piece. err: <nil>
Dec 30 08:30:39.604934093 (593h1m39.476017138s) [seg_pos={0 0}, upload=519885]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:30:39.605121220 (593h1m39.476204255s) [noise=true , address=45.91.116.12:4227    , piece_id=J7FQHX5P2AVKDJYKX7ZFIKNJAGAQIDZJZS7QSDAELKX262YZRMKQ, seg_pos={0 0}, upload=519885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Dec 30 08:30:39.605128543 (593h1m39.476211588s) [noise=true , address=45.91.116.12:4227    , piece_id=J7FQHX5P2AVKDJYKX7ZFIKNJAGAQIDZJZS7QSDAELKX262YZRMKQ, seg_pos={0 0}, upload=519885]: Long tail cancellation due to optimal threshold
Dec 30 08:30:39.605315168 (593h1m39.476398213s) [noise=true , address=77.90.31.15:4697     , piece_id=654JXMB62N2NJNF6DNKAG3QMNM7JHE3HWZ3IEVPFEK3PH6YTLLGQ, seg_pos={0 0}, upload=519885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:53780->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:53780->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53780->77.90.31.15:4697: use of closed network connection
Dec 30 08:30:39.605324517 (593h1m39.476407552s) [noise=true , address=77.90.31.15:4697     , piece_id=654JXMB62N2NJNF6DNKAG3QMNM7JHE3HWZ3IEVPFEK3PH6YTLLGQ, seg_pos={0 0}, upload=519885]: Long tail cancellation due to optimal threshold
Dec 30 08:30:39.605368731 (593h1m39.476451776s) [noise=true , address=77.90.31.19:4841     , piece_id=SVO4MSXS5TBEUNX4QQM6LNQRKPEW4THRFOAQMPTWKVAN5LBZWWVA, seg_pos={0 0}, upload=519885]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:46948->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 08:30:39.605375113 (593h1m39.476458158s) [noise=true , address=77.90.31.19:4841     , piece_id=SVO4MSXS5TBEUNX4QQM6LNQRKPEW4THRFOAQMPTWKVAN5LBZWWVA, seg_pos={0 0}, upload=519885]: Long tail cancellation due to optimal threshold
Dec 30 08:30:39.605390543 (593h1m39.476473588s) [seg_pos={0 0}, upload=519885]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:30:39.605456227 (593h1m39.476539262s) [upload=519885]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:30:39.605460296 (593h1m39.476543341s) [upload=519885]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:30:39.898389291 (593h1m39.769472336s) [upload=519885]: Done uploading segments...
Dec 30 08:30:39.898415551 (593h1m39.769498586s) [upload=519885]: Upload finished. err: <nil>
Duration: 30.003842433s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
13 out of 60 is failed (threshold=6) [X.XXXXXX.XX.......X..X..X..................X................]
duration: 
3.635275268s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:31:01 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.6353185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............X................X............................]
duration: 
48.263847535s
trace: 
https://jaeger.dev.storj.io/trace/3368aa43178d2f5a
FAIL (2025-12-30 08:30:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment
Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments...
Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982   , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289     , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766   , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection
Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection
Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection
Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments...
Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil>
Duration: 48.263864057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [.............X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X.]
duration: 
6.12328801s
trace: 
https://jaeger.dev.storj.io/trace/5f73013d6804d5dc
FAIL (2025-12-30 08:30:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:29:01.037510324 (593h0m0.908593359s) [upload=519865]: Starting upload
Dec 30 08:29:01.037599364 (593h0m0.908682409s) [upload=519865]: Done starting upload
Dec 30 08:29:01.037698363 (593h0m0.908781408s) [upload=519865]: Uploading segments...
Dec 30 08:29:01.037913092 (593h0m0.908996127s) [upload=519865]: Got next segment. Inline: false
Dec 30 08:29:01.037940754 (593h0m0.90902379s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:29:01.037945063 (593h0m0.909028108s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:29:01.283536410 (593h0m1.154619465s) [seg_pos={0 0}, upload=519865]: Begin upload segment...
Dec 30 08:29:01.283620299 (593h0m1.154703344s) [seg_pos={0 0}, upload=519865]: Done begin upload segment.
Dec 30 08:29:01.283641760 (593h0m1.154724795s) [noise=true , address=145.79.227.8:4331    , piece_id=E6P6WWQXAJPNIJMF3IZEOHMEQ7ZUZ2FLXGV6EO7QGBVOBA5W65JA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.283803138 (593h0m1.154886183s) [noise=true , address=145.79.227.42:4123   , piece_id=56LDFJIF27K3X57OCGTZQ73CIQJL7JPMAP25IWOML6DROIWW7YMQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.284306145 (593h0m1.155389191s) [noise=true , address=45.91.116.5:4300     , piece_id=5BFL5SOPUXMT5LOM7QQHC3TQLNUXB3F3HTTEONUA7HUJJDSUVFLA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.285558921 (593h0m1.156641966s) [noise=true , address=145.79.227.32:4834   , piece_id=V6K2IQWTBY74V5JANL6OEQPXN3QQX3ZCTPMUEOLJD4RTK4UJX6DQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.286087648 (593h0m1.157170693s) [noise=true , address=145.79.227.48:4301   , piece_id=U256FSWDESVEAOEIBJQR6V44AGGHZYQIVSMHN3TH6C5BYHMJIOZQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.286557543 (593h0m1.157640588s) [noise=true , address=145.79.227.16:4911   , piece_id=5XQKTXB6SCXAD2T7XL3CFSWRBO64EY3C5TRNVDAIAPSC6VW6UISA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.287049159 (593h0m1.158132204s) [noise=true , address=145.79.227.20:4437   , piece_id=ECG77AYMBUSZCAVT6FY3PWVNWGWTGGVNUH5UJYQ6QOBUHHSFUKUA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.288276956 (593h0m1.159360002s) [noise=true , address=145.79.227.50:4375   , piece_id=STXQ3PSKRVTUHQBYFKZ7DWXK2TRID2CBPY34OBWX46FX3566RLNQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.289832828 (593h0m1.160915874s) [noise=true , address=145.79.227.21:4473   , piece_id=DD7GTTDXUSF4Y2CSW3RHIEZFCBRE7LV7GOKTVXBRNCP22GNB6IYQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.290455404 (593h0m1.161538449s) [noise=true , address=77.90.31.21:4880     , piece_id=STN7GOQ7B3PKRB5YCHBXYQUKKCXCVI72763LPMXSNWGADAXVVCFA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.291694614 (593h0m1.162777659s) [noise=true , address=145.79.227.6:4257    , piece_id=IOZKI7R7MO6V62GY6WG332OPAKP7A7KJI43FRGJLZAFKCIDXZGMA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.292865142 (593h0m1.163948197s) [noise=true , address=77.90.31.16:4735     , piece_id=O2CLGHO5IOSR5AOPSEQEW5VZKR3E2GJGNYGUH46KUTMAQ2LYHRDA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.293498379 (593h0m1.164581444s) [noise=true , address=145.79.227.9:4366    , piece_id=KND5VFPR52TO4X4LBR7QGWK24TD3CIFO7M4B3NMAZOVD2BHNEPGA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.294989047 (593h0m1.166072092s) [noise=true , address=45.91.116.8:4048     , piece_id=KVVRUVRXEQZBJYVJMB4JFKXWIWGFU323T6RYQWQE4LFIOX4UKSQA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.297431458 (593h0m1.168514493s) [noise=true , address=77.90.31.15:4700     , piece_id=MAE2TBWZ6EQKSDJ6FQIF3AWZZOTF37V6QSESGBTHXP44GQZSWCIQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.301182149 (593h0m1.172265195s) [noise=true , address=145.79.227.46:4603   , piece_id=KQEEWCSOAFX4QI3MCTVSQQQXBERQCIBVUZDDNMN3UHV7INXBFQYQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:03.016739110 (593h0m2.887822155s) [upload=519865]: Next returned nil segment
Dec 30 08:29:03.016756903 (593h0m2.887839948s) [upload=519865]: Waiting for error group managing segments...
Dec 30 08:29:03.293753581 (593h0m3.164836646s) [noise=true , address=145.79.227.32:4834   , piece_id=V6K2IQWTBY74V5JANL6OEQPXN3QQX3ZCTPMUEOLJD4RTK4UJX6DQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.295164828 (593h0m3.166247873s) [noise=true , address=145.79.227.50:4375   , piece_id=STXQ3PSKRVTUHQBYFKZ7DWXK2TRID2CBPY34OBWX46FX3566RLNQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.301232392 (593h0m3.172315438s) [noise=true , address=145.79.227.20:4437   , piece_id=ECG77AYMBUSZCAVT6FY3PWVNWGWTGGVNUH5UJYQ6QOBUHHSFUKUA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.305901734 (593h0m3.176984769s) [noise=true , address=145.79.227.48:4301   , piece_id=U256FSWDESVEAOEIBJQR6V44AGGHZYQIVSMHN3TH6C5BYHMJIOZQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.308473411 (593h0m3.179556456s) [noise=true , address=145.79.227.42:4123   , piece_id=56LDFJIF27K3X57OCGTZQ73CIQJL7JPMAP25IWOML6DROIWW7YMQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.319073785 (593h0m3.19015683s) [noise=true , address=145.79.227.21:4473   , piece_id=DD7GTTDXUSF4Y2CSW3RHIEZFCBRE7LV7GOKTVXBRNCP22GNB6IYQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.324987746 (593h0m3.196070791s) [noise=true , address=145.79.227.9:4366    , piece_id=KND5VFPR52TO4X4LBR7QGWK24TD3CIFO7M4B3NMAZOVD2BHNEPGA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.332820611 (593h0m3.203903657s) [noise=true , address=145.79.227.46:4603   , piece_id=KQEEWCSOAFX4QI3MCTVSQQQXBERQCIBVUZDDNMN3UHV7INXBFQYQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.348324484 (593h0m3.219407529s) [noise=true , address=145.79.227.6:4257    , piece_id=IOZKI7R7MO6V62GY6WG332OPAKP7A7KJI43FRGJLZAFKCIDXZGMA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.391177633 (593h0m3.262260678s) [noise=true , address=145.79.227.8:4331    , piece_id=E6P6WWQXAJPNIJMF3IZEOHMEQ7ZUZ2FLXGV6EO7QGBVOBA5W65JA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.481081536 (593h0m3.352164581s) [noise=true , address=77.90.31.15:4700     , piece_id=MAE2TBWZ6EQKSDJ6FQIF3AWZZOTF37V6QSESGBTHXP44GQZSWCIQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.553675246 (593h0m3.424758281s) [noise=true , address=77.90.31.21:4880     , piece_id=STN7GOQ7B3PKRB5YCHBXYQUKKCXCVI72763LPMXSNWGADAXVVCFA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.632143152 (593h0m3.503226207s) [noise=true , address=45.91.116.5:4300     , piece_id=5BFL5SOPUXMT5LOM7QQHC3TQLNUXB3F3HTTEONUA7HUJJDSUVFLA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.632174391 (593h0m3.503257436s) [seg_pos={0 0}, upload=519865]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:29:03.632562240 (593h0m3.503645305s) [noise=true , address=77.90.31.16:4735     , piece_id=O2CLGHO5IOSR5AOPSEQEW5VZKR3E2GJGNYGUH46KUTMAQ2LYHRDA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58734->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58734->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58734->77.90.31.16:4735: use of closed network connection
Dec 30 08:29:03.632573772 (593h0m3.503656837s) [noise=true , address=77.90.31.16:4735     , piece_id=O2CLGHO5IOSR5AOPSEQEW5VZKR3E2GJGNYGUH46KUTMAQ2LYHRDA, seg_pos={0 0}, upload=519865]: Long tail cancellation due to optimal threshold
Dec 30 08:29:03.632922426 (593h0m3.504005471s) [noise=true , address=145.79.227.16:4911   , piece_id=5XQKTXB6SCXAD2T7XL3CFSWRBO64EY3C5TRNVDAIAPSC6VW6UISA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF
Dec 30 08:29:03.632955048 (593h0m3.504038093s) [noise=true , address=145.79.227.16:4911   , piece_id=5XQKTXB6SCXAD2T7XL3CFSWRBO64EY3C5TRNVDAIAPSC6VW6UISA, seg_pos={0 0}, upload=519865]: Long tail cancellation due to optimal threshold
Dec 30 08:29:03.632993401 (593h0m3.504076456s) [noise=true , address=45.91.116.8:4048     , piece_id=KVVRUVRXEQZBJYVJMB4JFKXWIWGFU323T6RYQWQE4LFIOX4UKSQA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:39670->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 08:29:03.633046472 (593h0m3.504129517s) [noise=true , address=45.91.116.8:4048     , piece_id=KVVRUVRXEQZBJYVJMB4JFKXWIWGFU323T6RYQWQE4LFIOX4UKSQA, seg_pos={0 0}, upload=519865]: Long tail cancellation due to optimal threshold
Dec 30 08:29:03.633164036 (593h0m3.504247081s) [seg_pos={0 0}, upload=519865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:29:03.633282371 (593h0m3.504365406s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:29:03.633288392 (593h0m3.504371438s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:29:03.939864035 (593h0m3.81094708s) [upload=519865]: Done uploading segments...
Dec 30 08:29:03.939889113 (593h0m3.810972158s) [upload=519865]: Upload finished. err: <nil>
Duration: 6.123319782s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
13 out of 60 is failed (threshold=6) [.X.XXXXXX.XX.......X..X..X..................X...............]
duration: 
4.166164488s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:29:49 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.166190447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X................X...........................]
duration: 
34.779243092s
trace: 
https://jaeger.dev.storj.io/trace/3368aa43178d2f5a
FAIL (2025-12-30 08:29:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment
Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments...
Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982   , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289     , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766   , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection
Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection
Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection
Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments...
Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil>
Duration: 34.779276083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [..............X...X..X..X..XX.X.X...XX.XX...X...X...X.X.X..X]
duration: 
30.003590734s
trace: 
https://jaeger.dev.storj.io/trace/5f73013d6804d5dc
FAIL (2025-12-30 08:29:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 08:29:01.037510324 (593h0m0.908593359s) [upload=519865]: Starting upload
Dec 30 08:29:01.037599364 (593h0m0.908682409s) [upload=519865]: Done starting upload
Dec 30 08:29:01.037698363 (593h0m0.908781408s) [upload=519865]: Uploading segments...
Dec 30 08:29:01.037913092 (593h0m0.908996127s) [upload=519865]: Got next segment. Inline: false
Dec 30 08:29:01.037940754 (593h0m0.90902379s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 08:29:01.037945063 (593h0m0.909028108s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 08:29:01.283536410 (593h0m1.154619465s) [seg_pos={0 0}, upload=519865]: Begin upload segment...
Dec 30 08:29:01.283620299 (593h0m1.154703344s) [seg_pos={0 0}, upload=519865]: Done begin upload segment.
Dec 30 08:29:01.283641760 (593h0m1.154724795s) [noise=true , address=145.79.227.8:4331    , piece_id=E6P6WWQXAJPNIJMF3IZEOHMEQ7ZUZ2FLXGV6EO7QGBVOBA5W65JA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.283803138 (593h0m1.154886183s) [noise=true , address=145.79.227.42:4123   , piece_id=56LDFJIF27K3X57OCGTZQ73CIQJL7JPMAP25IWOML6DROIWW7YMQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.284306145 (593h0m1.155389191s) [noise=true , address=45.91.116.5:4300     , piece_id=5BFL5SOPUXMT5LOM7QQHC3TQLNUXB3F3HTTEONUA7HUJJDSUVFLA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.285558921 (593h0m1.156641966s) [noise=true , address=145.79.227.32:4834   , piece_id=V6K2IQWTBY74V5JANL6OEQPXN3QQX3ZCTPMUEOLJD4RTK4UJX6DQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.286087648 (593h0m1.157170693s) [noise=true , address=145.79.227.48:4301   , piece_id=U256FSWDESVEAOEIBJQR6V44AGGHZYQIVSMHN3TH6C5BYHMJIOZQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.286557543 (593h0m1.157640588s) [noise=true , address=145.79.227.16:4911   , piece_id=5XQKTXB6SCXAD2T7XL3CFSWRBO64EY3C5TRNVDAIAPSC6VW6UISA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.287049159 (593h0m1.158132204s) [noise=true , address=145.79.227.20:4437   , piece_id=ECG77AYMBUSZCAVT6FY3PWVNWGWTGGVNUH5UJYQ6QOBUHHSFUKUA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.288276956 (593h0m1.159360002s) [noise=true , address=145.79.227.50:4375   , piece_id=STXQ3PSKRVTUHQBYFKZ7DWXK2TRID2CBPY34OBWX46FX3566RLNQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.289832828 (593h0m1.160915874s) [noise=true , address=145.79.227.21:4473   , piece_id=DD7GTTDXUSF4Y2CSW3RHIEZFCBRE7LV7GOKTVXBRNCP22GNB6IYQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.290455404 (593h0m1.161538449s) [noise=true , address=77.90.31.21:4880     , piece_id=STN7GOQ7B3PKRB5YCHBXYQUKKCXCVI72763LPMXSNWGADAXVVCFA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.291694614 (593h0m1.162777659s) [noise=true , address=145.79.227.6:4257    , piece_id=IOZKI7R7MO6V62GY6WG332OPAKP7A7KJI43FRGJLZAFKCIDXZGMA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.292865142 (593h0m1.163948197s) [noise=true , address=77.90.31.16:4735     , piece_id=O2CLGHO5IOSR5AOPSEQEW5VZKR3E2GJGNYGUH46KUTMAQ2LYHRDA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.293498379 (593h0m1.164581444s) [noise=true , address=145.79.227.9:4366    , piece_id=KND5VFPR52TO4X4LBR7QGWK24TD3CIFO7M4B3NMAZOVD2BHNEPGA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.294989047 (593h0m1.166072092s) [noise=true , address=45.91.116.8:4048     , piece_id=KVVRUVRXEQZBJYVJMB4JFKXWIWGFU323T6RYQWQE4LFIOX4UKSQA, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.297431458 (593h0m1.168514493s) [noise=true , address=77.90.31.15:4700     , piece_id=MAE2TBWZ6EQKSDJ6FQIF3AWZZOTF37V6QSESGBTHXP44GQZSWCIQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:01.301182149 (593h0m1.172265195s) [noise=true , address=145.79.227.46:4603   , piece_id=KQEEWCSOAFX4QI3MCTVSQQQXBERQCIBVUZDDNMN3UHV7INXBFQYQ, seg_pos={0 0}, upload=519865]: Uploading piece...
Dec 30 08:29:03.016739110 (593h0m2.887822155s) [upload=519865]: Next returned nil segment
Dec 30 08:29:03.016756903 (593h0m2.887839948s) [upload=519865]: Waiting for error group managing segments...
Dec 30 08:29:03.293753581 (593h0m3.164836646s) [noise=true , address=145.79.227.32:4834   , piece_id=V6K2IQWTBY74V5JANL6OEQPXN3QQX3ZCTPMUEOLJD4RTK4UJX6DQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.295164828 (593h0m3.166247873s) [noise=true , address=145.79.227.50:4375   , piece_id=STXQ3PSKRVTUHQBYFKZ7DWXK2TRID2CBPY34OBWX46FX3566RLNQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.301232392 (593h0m3.172315438s) [noise=true , address=145.79.227.20:4437   , piece_id=ECG77AYMBUSZCAVT6FY3PWVNWGWTGGVNUH5UJYQ6QOBUHHSFUKUA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.305901734 (593h0m3.176984769s) [noise=true , address=145.79.227.48:4301   , piece_id=U256FSWDESVEAOEIBJQR6V44AGGHZYQIVSMHN3TH6C5BYHMJIOZQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.308473411 (593h0m3.179556456s) [noise=true , address=145.79.227.42:4123   , piece_id=56LDFJIF27K3X57OCGTZQ73CIQJL7JPMAP25IWOML6DROIWW7YMQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.319073785 (593h0m3.19015683s) [noise=true , address=145.79.227.21:4473   , piece_id=DD7GTTDXUSF4Y2CSW3RHIEZFCBRE7LV7GOKTVXBRNCP22GNB6IYQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.324987746 (593h0m3.196070791s) [noise=true , address=145.79.227.9:4366    , piece_id=KND5VFPR52TO4X4LBR7QGWK24TD3CIFO7M4B3NMAZOVD2BHNEPGA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.332820611 (593h0m3.203903657s) [noise=true , address=145.79.227.46:4603   , piece_id=KQEEWCSOAFX4QI3MCTVSQQQXBERQCIBVUZDDNMN3UHV7INXBFQYQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.348324484 (593h0m3.219407529s) [noise=true , address=145.79.227.6:4257    , piece_id=IOZKI7R7MO6V62GY6WG332OPAKP7A7KJI43FRGJLZAFKCIDXZGMA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.391177633 (593h0m3.262260678s) [noise=true , address=145.79.227.8:4331    , piece_id=E6P6WWQXAJPNIJMF3IZEOHMEQ7ZUZ2FLXGV6EO7QGBVOBA5W65JA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.481081536 (593h0m3.352164581s) [noise=true , address=77.90.31.15:4700     , piece_id=MAE2TBWZ6EQKSDJ6FQIF3AWZZOTF37V6QSESGBTHXP44GQZSWCIQ, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.553675246 (593h0m3.424758281s) [noise=true , address=77.90.31.21:4880     , piece_id=STN7GOQ7B3PKRB5YCHBXYQUKKCXCVI72763LPMXSNWGADAXVVCFA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.632143152 (593h0m3.503226207s) [noise=true , address=45.91.116.5:4300     , piece_id=5BFL5SOPUXMT5LOM7QQHC3TQLNUXB3F3HTTEONUA7HUJJDSUVFLA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: <nil>
Dec 30 08:29:03.632174391 (593h0m3.503257436s) [seg_pos={0 0}, upload=519865]: Segment reached optimal threshold of 13 pieces.
Dec 30 08:29:03.632562240 (593h0m3.503645305s) [noise=true , address=77.90.31.16:4735     , piece_id=O2CLGHO5IOSR5AOPSEQEW5VZKR3E2GJGNYGUH46KUTMAQ2LYHRDA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:58734->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58734->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58734->77.90.31.16:4735: use of closed network connection
Dec 30 08:29:03.632573772 (593h0m3.503656837s) [noise=true , address=77.90.31.16:4735     , piece_id=O2CLGHO5IOSR5AOPSEQEW5VZKR3E2GJGNYGUH46KUTMAQ2LYHRDA, seg_pos={0 0}, upload=519865]: Long tail cancellation due to optimal threshold
Dec 30 08:29:03.632922426 (593h0m3.504005471s) [noise=true , address=145.79.227.16:4911   , piece_id=5XQKTXB6SCXAD2T7XL3CFSWRBO64EY3C5TRNVDAIAPSC6VW6UISA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF
Dec 30 08:29:03.632955048 (593h0m3.504038093s) [noise=true , address=145.79.227.16:4911   , piece_id=5XQKTXB6SCXAD2T7XL3CFSWRBO64EY3C5TRNVDAIAPSC6VW6UISA, seg_pos={0 0}, upload=519865]: Long tail cancellation due to optimal threshold
Dec 30 08:29:03.632993401 (593h0m3.504076456s) [noise=true , address=45.91.116.8:4048     , piece_id=KVVRUVRXEQZBJYVJMB4JFKXWIWGFU323T6RYQWQE4LFIOX4UKSQA, seg_pos={0 0}, upload=519865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:39670->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 08:29:03.633046472 (593h0m3.504129517s) [noise=true , address=45.91.116.8:4048     , piece_id=KVVRUVRXEQZBJYVJMB4JFKXWIWGFU323T6RYQWQE4LFIOX4UKSQA, seg_pos={0 0}, upload=519865]: Long tail cancellation due to optimal threshold
Dec 30 08:29:03.633164036 (593h0m3.504247081s) [seg_pos={0 0}, upload=519865]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 08:29:03.633282371 (593h0m3.504365406s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:29:03.633288392 (593h0m3.504371438s) [upload=519865]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:29:03.939864035 (593h0m3.81094708s) [upload=519865]: Done uploading segments...
Dec 30 08:29:03.939889113 (593h0m3.810972158s) [upload=519865]: Upload finished. err: <nil>
Duration: 30.00361525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
14 out of 60 is failed (threshold=6) [X.X.XXXXXX.XX.......X..X..X..................X..............]
duration: 
3.64187857s
trace: 
https://jaeger.dev.storj.io/trace/7a032e32a5763d1a
FAIL (2025-12-30 08:28:46 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.641902034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................X................X..........................]
duration: 
42.245287041s
trace: 
https://jaeger.dev.storj.io/trace/3368aa43178d2f5a
FAIL (2025-12-30 08:28:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece...
Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment
Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments...
Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298    , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604   , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965     , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982   , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949     , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856     , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289     , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576     , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766   , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902     , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219   , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935   , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil>
Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces.
Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection
Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654     , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection
Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573     , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF
Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190   , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection
Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707    , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold
Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments...
Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil>
Duration: 42.245315466s core uplink us-select-1 us1-placement-20