Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 902109 reports (Page 11314 of 18043)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [.............X...X.X...X..X..X.X..X..X.......X...........X..]
duration: 
6.051763682s
trace: 
https://jaeger.dev.storj.io/trace/55d95903079356b1
FAIL (2025-12-29 00:43:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:41:49.783915674 (561h12m49.654998749s) [upload=492031]: Starting upload
Dec 29 00:41:49.784013711 (561h12m49.655096756s) [upload=492031]: Done starting upload
Dec 29 00:41:49.784072261 (561h12m49.655155297s) [upload=492031]: Uploading segments...
Dec 29 00:41:49.784286150 (561h12m49.655369195s) [upload=492031]: Got next segment. Inline: false
Dec 29 00:41:49.784333439 (561h12m49.655416484s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:41:49.784338689 (561h12m49.655421734s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:41:50.028538026 (561h12m49.899621061s) [seg_pos={0 0}, upload=492031]: Begin upload segment...
Dec 29 00:41:50.028583141 (561h12m49.899666176s) [seg_pos={0 0}, upload=492031]: Done begin upload segment.
Dec 29 00:41:50.028608239 (561h12m49.899691285s) [noise=true , address=45.91.116.12:4227    , piece_id=4U63YQR5FDDY735HXH6W26ATKH4Z6ASBFUOQS72OBFAXNJLRRVUQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.028790516 (561h12m49.899873561s) [noise=true , address=145.79.227.45:4229   , piece_id=TXFF2Z6H4PWWCTZFDH5YTYOO2F26M7S7U2YFGIHS6RBWS3WPEE7A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029231106 (561h12m49.900314141s) [noise=true , address=145.79.227.52:4445   , piece_id=P7E6RRMQ3RMKAUBU4DPWOBQYKXUZFPUADTEG6GO7LJ6VD3QIDZ4A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029501411 (561h12m49.900584446s) [noise=true , address=77.90.31.13:4629     , piece_id=YU6DEIYH4KVZE3YCTXWMV2I2LBB2LXDMORMD5IFU6SDPSPOQ3ARA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029776925 (561h12m49.90085997s) [noise=true , address=145.79.227.2:4114    , piece_id=GIDC4Z4FLBKHAQUJVMOD7GIB4GFMUVV6ITEG43RVE7M5ZH42PQ6A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030190653 (561h12m49.901273688s) [noise=true , address=77.90.31.14:4662     , piece_id=2GTP6I75HR2KIFL35MPCV5T3GU7AXTZ3BAERZGU6CTSU2UIMZJQA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030474113 (561h12m49.901557148s) [noise=true , address=145.79.227.50:4373   , piece_id=UDX6TREA5ZV2ZBAOIIY7EWXZR3HZ7GV6ZO5PKMXDUDD4YCJBXNCQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030765547 (561h12m49.901848582s) [noise=true , address=77.90.31.11:4554     , piece_id=EHGKDAWWHT5XURIH22FIFAI53FUGX6WHRFELZUHA7XSS25AYNRRA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.031131134 (561h12m49.902214179s) [noise=true , address=45.91.116.10:4083    , piece_id=T5KWFMNVVGAII5NCQN4WFLO3HTK5NUASSKD2HL36T3H44LTFMI4Q, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.032118874 (561h12m49.90320191s) [noise=true , address=45.91.116.8:4046     , piece_id=UXAGDZGUC5AORPCUC3RG42KSL726ZZ6KQSIUIU3JZMPNBEPVJXQA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.034388266 (561h12m49.905471311s) [noise=true , address=145.79.227.11:4730   , piece_id=2JN5ANVFOYXTUMCDS3QLNSGCWHSWJFANBK3ZSPXJ235KTHIIOBSQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.037936692 (561h12m49.909019738s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.040716356 (561h12m49.911799401s) [noise=true , address=145.79.227.4:4186    , piece_id=SZM2R5WN7HFHKCVTIS36XBGJ7I2GPIYLJFWBBIZR4LA4LL7ULSCA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.040868725 (561h12m49.91195176s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.041949243 (561h12m49.913032278s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.045032964 (561h12m49.916116009s) [noise=true , address=45.91.116.7:4010     , piece_id=KWW6ZY7YFBKRVZIJBOKECHZ3QOQRL2IW2Y2CFXEW2KHAACKFA7QQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:51.733636439 (561h12m51.604719484s) [upload=492031]: Next returned nil segment
Dec 29 00:41:51.733721371 (561h12m51.604804426s) [upload=492031]: Waiting for error group managing segments...
Dec 29 00:41:52.023793254 (561h12m51.894876299s) [noise=true , address=145.79.227.45:4229   , piece_id=TXFF2Z6H4PWWCTZFDH5YTYOO2F26M7S7U2YFGIHS6RBWS3WPEE7A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.031612082 (561h12m51.902695127s) [noise=true , address=145.79.227.50:4373   , piece_id=UDX6TREA5ZV2ZBAOIIY7EWXZR3HZ7GV6ZO5PKMXDUDD4YCJBXNCQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.031658311 (561h12m51.902741346s) [noise=true , address=145.79.227.52:4445   , piece_id=P7E6RRMQ3RMKAUBU4DPWOBQYKXUZFPUADTEG6GO7LJ6VD3QIDZ4A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.034878913 (561h12m51.905961948s) [noise=true , address=145.79.227.2:4114    , piece_id=GIDC4Z4FLBKHAQUJVMOD7GIB4GFMUVV6ITEG43RVE7M5ZH42PQ6A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.040500507 (561h12m51.911583552s) [noise=true , address=145.79.227.11:4730   , piece_id=2JN5ANVFOYXTUMCDS3QLNSGCWHSWJFANBK3ZSPXJ235KTHIIOBSQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.069432146 (561h12m51.940515191s) [noise=true , address=145.79.227.4:4186    , piece_id=SZM2R5WN7HFHKCVTIS36XBGJ7I2GPIYLJFWBBIZR4LA4LL7ULSCA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.234775952 (561h12m52.105858997s) [noise=true , address=77.90.31.13:4629     , piece_id=YU6DEIYH4KVZE3YCTXWMV2I2LBB2LXDMORMD5IFU6SDPSPOQ3ARA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.328030110 (561h12m52.199113145s) [noise=true , address=45.91.116.12:4227    , piece_id=4U63YQR5FDDY735HXH6W26ATKH4Z6ASBFUOQS72OBFAXNJLRRVUQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.330336822 (561h12m52.201419857s) [noise=true , address=77.90.31.14:4662     , piece_id=2GTP6I75HR2KIFL35MPCV5T3GU7AXTZ3BAERZGU6CTSU2UIMZJQA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.341867358 (561h12m52.212950403s) [noise=true , address=45.91.116.7:4010     , piece_id=KWW6ZY7YFBKRVZIJBOKECHZ3QOQRL2IW2Y2CFXEW2KHAACKFA7QQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.386681009 (561h12m52.257764054s) [noise=true , address=45.91.116.10:4083    , piece_id=T5KWFMNVVGAII5NCQN4WFLO3HTK5NUASSKD2HL36T3H44LTFMI4Q, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.387083865 (561h12m52.25816691s) [noise=true , address=45.91.116.8:4046     , piece_id=UXAGDZGUC5AORPCUC3RG42KSL726ZZ6KQSIUIU3JZMPNBEPVJXQA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.952947611 (561h12m52.824030656s) [noise=true , address=77.90.31.11:4554     , piece_id=EHGKDAWWHT5XURIH22FIFAI53FUGX6WHRFELZUHA7XSS25AYNRRA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.952971467 (561h12m52.824054522s) [seg_pos={0 0}, upload=492031]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:41:52.953201746 (561h12m52.824284781s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953209971 (561h12m52.824293016s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953326773 (561h12m52.824409848s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953350198 (561h12m52.824433243s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953474654 (561h12m52.824557699s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953484374 (561h12m52.824567419s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953505583 (561h12m52.824588628s) [seg_pos={0 0}, upload=492031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:41:52.953611595 (561h12m52.82469464s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:41:52.953617466 (561h12m52.824700521s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:41:53.239743086 (561h12m53.110826121s) [upload=492031]: Done uploading segments...
Dec 29 00:41:53.239762553 (561h12m53.110845588s) [upload=492031]: Upload finished. err: <nil>
Duration: 6.051810831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.........X...X........X.X........X.X..........X.............]
duration: 
15.158368466s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:43:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 15.158392893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [..............X...X.X...X..X..X.X..X..X.......X...........X.]
duration: 
6.275226153s
trace: 
https://jaeger.dev.storj.io/trace/55d95903079356b1
FAIL (2025-12-29 00:42:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:41:49.783915674 (561h12m49.654998749s) [upload=492031]: Starting upload
Dec 29 00:41:49.784013711 (561h12m49.655096756s) [upload=492031]: Done starting upload
Dec 29 00:41:49.784072261 (561h12m49.655155297s) [upload=492031]: Uploading segments...
Dec 29 00:41:49.784286150 (561h12m49.655369195s) [upload=492031]: Got next segment. Inline: false
Dec 29 00:41:49.784333439 (561h12m49.655416484s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:41:49.784338689 (561h12m49.655421734s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:41:50.028538026 (561h12m49.899621061s) [seg_pos={0 0}, upload=492031]: Begin upload segment...
Dec 29 00:41:50.028583141 (561h12m49.899666176s) [seg_pos={0 0}, upload=492031]: Done begin upload segment.
Dec 29 00:41:50.028608239 (561h12m49.899691285s) [noise=true , address=45.91.116.12:4227    , piece_id=4U63YQR5FDDY735HXH6W26ATKH4Z6ASBFUOQS72OBFAXNJLRRVUQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.028790516 (561h12m49.899873561s) [noise=true , address=145.79.227.45:4229   , piece_id=TXFF2Z6H4PWWCTZFDH5YTYOO2F26M7S7U2YFGIHS6RBWS3WPEE7A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029231106 (561h12m49.900314141s) [noise=true , address=145.79.227.52:4445   , piece_id=P7E6RRMQ3RMKAUBU4DPWOBQYKXUZFPUADTEG6GO7LJ6VD3QIDZ4A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029501411 (561h12m49.900584446s) [noise=true , address=77.90.31.13:4629     , piece_id=YU6DEIYH4KVZE3YCTXWMV2I2LBB2LXDMORMD5IFU6SDPSPOQ3ARA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029776925 (561h12m49.90085997s) [noise=true , address=145.79.227.2:4114    , piece_id=GIDC4Z4FLBKHAQUJVMOD7GIB4GFMUVV6ITEG43RVE7M5ZH42PQ6A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030190653 (561h12m49.901273688s) [noise=true , address=77.90.31.14:4662     , piece_id=2GTP6I75HR2KIFL35MPCV5T3GU7AXTZ3BAERZGU6CTSU2UIMZJQA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030474113 (561h12m49.901557148s) [noise=true , address=145.79.227.50:4373   , piece_id=UDX6TREA5ZV2ZBAOIIY7EWXZR3HZ7GV6ZO5PKMXDUDD4YCJBXNCQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030765547 (561h12m49.901848582s) [noise=true , address=77.90.31.11:4554     , piece_id=EHGKDAWWHT5XURIH22FIFAI53FUGX6WHRFELZUHA7XSS25AYNRRA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.031131134 (561h12m49.902214179s) [noise=true , address=45.91.116.10:4083    , piece_id=T5KWFMNVVGAII5NCQN4WFLO3HTK5NUASSKD2HL36T3H44LTFMI4Q, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.032118874 (561h12m49.90320191s) [noise=true , address=45.91.116.8:4046     , piece_id=UXAGDZGUC5AORPCUC3RG42KSL726ZZ6KQSIUIU3JZMPNBEPVJXQA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.034388266 (561h12m49.905471311s) [noise=true , address=145.79.227.11:4730   , piece_id=2JN5ANVFOYXTUMCDS3QLNSGCWHSWJFANBK3ZSPXJ235KTHIIOBSQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.037936692 (561h12m49.909019738s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.040716356 (561h12m49.911799401s) [noise=true , address=145.79.227.4:4186    , piece_id=SZM2R5WN7HFHKCVTIS36XBGJ7I2GPIYLJFWBBIZR4LA4LL7ULSCA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.040868725 (561h12m49.91195176s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.041949243 (561h12m49.913032278s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.045032964 (561h12m49.916116009s) [noise=true , address=45.91.116.7:4010     , piece_id=KWW6ZY7YFBKRVZIJBOKECHZ3QOQRL2IW2Y2CFXEW2KHAACKFA7QQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:51.733636439 (561h12m51.604719484s) [upload=492031]: Next returned nil segment
Dec 29 00:41:51.733721371 (561h12m51.604804426s) [upload=492031]: Waiting for error group managing segments...
Dec 29 00:41:52.023793254 (561h12m51.894876299s) [noise=true , address=145.79.227.45:4229   , piece_id=TXFF2Z6H4PWWCTZFDH5YTYOO2F26M7S7U2YFGIHS6RBWS3WPEE7A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.031612082 (561h12m51.902695127s) [noise=true , address=145.79.227.50:4373   , piece_id=UDX6TREA5ZV2ZBAOIIY7EWXZR3HZ7GV6ZO5PKMXDUDD4YCJBXNCQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.031658311 (561h12m51.902741346s) [noise=true , address=145.79.227.52:4445   , piece_id=P7E6RRMQ3RMKAUBU4DPWOBQYKXUZFPUADTEG6GO7LJ6VD3QIDZ4A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.034878913 (561h12m51.905961948s) [noise=true , address=145.79.227.2:4114    , piece_id=GIDC4Z4FLBKHAQUJVMOD7GIB4GFMUVV6ITEG43RVE7M5ZH42PQ6A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.040500507 (561h12m51.911583552s) [noise=true , address=145.79.227.11:4730   , piece_id=2JN5ANVFOYXTUMCDS3QLNSGCWHSWJFANBK3ZSPXJ235KTHIIOBSQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.069432146 (561h12m51.940515191s) [noise=true , address=145.79.227.4:4186    , piece_id=SZM2R5WN7HFHKCVTIS36XBGJ7I2GPIYLJFWBBIZR4LA4LL7ULSCA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.234775952 (561h12m52.105858997s) [noise=true , address=77.90.31.13:4629     , piece_id=YU6DEIYH4KVZE3YCTXWMV2I2LBB2LXDMORMD5IFU6SDPSPOQ3ARA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.328030110 (561h12m52.199113145s) [noise=true , address=45.91.116.12:4227    , piece_id=4U63YQR5FDDY735HXH6W26ATKH4Z6ASBFUOQS72OBFAXNJLRRVUQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.330336822 (561h12m52.201419857s) [noise=true , address=77.90.31.14:4662     , piece_id=2GTP6I75HR2KIFL35MPCV5T3GU7AXTZ3BAERZGU6CTSU2UIMZJQA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.341867358 (561h12m52.212950403s) [noise=true , address=45.91.116.7:4010     , piece_id=KWW6ZY7YFBKRVZIJBOKECHZ3QOQRL2IW2Y2CFXEW2KHAACKFA7QQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.386681009 (561h12m52.257764054s) [noise=true , address=45.91.116.10:4083    , piece_id=T5KWFMNVVGAII5NCQN4WFLO3HTK5NUASSKD2HL36T3H44LTFMI4Q, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.387083865 (561h12m52.25816691s) [noise=true , address=45.91.116.8:4046     , piece_id=UXAGDZGUC5AORPCUC3RG42KSL726ZZ6KQSIUIU3JZMPNBEPVJXQA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.952947611 (561h12m52.824030656s) [noise=true , address=77.90.31.11:4554     , piece_id=EHGKDAWWHT5XURIH22FIFAI53FUGX6WHRFELZUHA7XSS25AYNRRA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.952971467 (561h12m52.824054522s) [seg_pos={0 0}, upload=492031]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:41:52.953201746 (561h12m52.824284781s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953209971 (561h12m52.824293016s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953326773 (561h12m52.824409848s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953350198 (561h12m52.824433243s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953474654 (561h12m52.824557699s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953484374 (561h12m52.824567419s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953505583 (561h12m52.824588628s) [seg_pos={0 0}, upload=492031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:41:52.953611595 (561h12m52.82469464s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:41:52.953617466 (561h12m52.824700521s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:41:53.239743086 (561h12m53.110826121s) [upload=492031]: Done uploading segments...
Dec 29 00:41:53.239762553 (561h12m53.110845588s) [upload=492031]: Upload finished. err: <nil>
Duration: 6.27525123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..........X...X........X.X........X.X..........X............]
duration: 
11.248100722s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:42:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 11.248128865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [...............X...X.X...X..X..X.X..X..X.......X...........X]
duration: 
30.003601624s
trace: 
https://jaeger.dev.storj.io/trace/55d95903079356b1
FAIL (2025-12-29 00:42:19 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:41:49.783915674 (561h12m49.654998749s) [upload=492031]: Starting upload
Dec 29 00:41:49.784013711 (561h12m49.655096756s) [upload=492031]: Done starting upload
Dec 29 00:41:49.784072261 (561h12m49.655155297s) [upload=492031]: Uploading segments...
Dec 29 00:41:49.784286150 (561h12m49.655369195s) [upload=492031]: Got next segment. Inline: false
Dec 29 00:41:49.784333439 (561h12m49.655416484s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:41:49.784338689 (561h12m49.655421734s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:41:50.028538026 (561h12m49.899621061s) [seg_pos={0 0}, upload=492031]: Begin upload segment...
Dec 29 00:41:50.028583141 (561h12m49.899666176s) [seg_pos={0 0}, upload=492031]: Done begin upload segment.
Dec 29 00:41:50.028608239 (561h12m49.899691285s) [noise=true , address=45.91.116.12:4227    , piece_id=4U63YQR5FDDY735HXH6W26ATKH4Z6ASBFUOQS72OBFAXNJLRRVUQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.028790516 (561h12m49.899873561s) [noise=true , address=145.79.227.45:4229   , piece_id=TXFF2Z6H4PWWCTZFDH5YTYOO2F26M7S7U2YFGIHS6RBWS3WPEE7A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029231106 (561h12m49.900314141s) [noise=true , address=145.79.227.52:4445   , piece_id=P7E6RRMQ3RMKAUBU4DPWOBQYKXUZFPUADTEG6GO7LJ6VD3QIDZ4A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029501411 (561h12m49.900584446s) [noise=true , address=77.90.31.13:4629     , piece_id=YU6DEIYH4KVZE3YCTXWMV2I2LBB2LXDMORMD5IFU6SDPSPOQ3ARA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.029776925 (561h12m49.90085997s) [noise=true , address=145.79.227.2:4114    , piece_id=GIDC4Z4FLBKHAQUJVMOD7GIB4GFMUVV6ITEG43RVE7M5ZH42PQ6A, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030190653 (561h12m49.901273688s) [noise=true , address=77.90.31.14:4662     , piece_id=2GTP6I75HR2KIFL35MPCV5T3GU7AXTZ3BAERZGU6CTSU2UIMZJQA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030474113 (561h12m49.901557148s) [noise=true , address=145.79.227.50:4373   , piece_id=UDX6TREA5ZV2ZBAOIIY7EWXZR3HZ7GV6ZO5PKMXDUDD4YCJBXNCQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.030765547 (561h12m49.901848582s) [noise=true , address=77.90.31.11:4554     , piece_id=EHGKDAWWHT5XURIH22FIFAI53FUGX6WHRFELZUHA7XSS25AYNRRA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.031131134 (561h12m49.902214179s) [noise=true , address=45.91.116.10:4083    , piece_id=T5KWFMNVVGAII5NCQN4WFLO3HTK5NUASSKD2HL36T3H44LTFMI4Q, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.032118874 (561h12m49.90320191s) [noise=true , address=45.91.116.8:4046     , piece_id=UXAGDZGUC5AORPCUC3RG42KSL726ZZ6KQSIUIU3JZMPNBEPVJXQA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.034388266 (561h12m49.905471311s) [noise=true , address=145.79.227.11:4730   , piece_id=2JN5ANVFOYXTUMCDS3QLNSGCWHSWJFANBK3ZSPXJ235KTHIIOBSQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.037936692 (561h12m49.909019738s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.040716356 (561h12m49.911799401s) [noise=true , address=145.79.227.4:4186    , piece_id=SZM2R5WN7HFHKCVTIS36XBGJ7I2GPIYLJFWBBIZR4LA4LL7ULSCA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.040868725 (561h12m49.91195176s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.041949243 (561h12m49.913032278s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:50.045032964 (561h12m49.916116009s) [noise=true , address=45.91.116.7:4010     , piece_id=KWW6ZY7YFBKRVZIJBOKECHZ3QOQRL2IW2Y2CFXEW2KHAACKFA7QQ, seg_pos={0 0}, upload=492031]: Uploading piece...
Dec 29 00:41:51.733636439 (561h12m51.604719484s) [upload=492031]: Next returned nil segment
Dec 29 00:41:51.733721371 (561h12m51.604804426s) [upload=492031]: Waiting for error group managing segments...
Dec 29 00:41:52.023793254 (561h12m51.894876299s) [noise=true , address=145.79.227.45:4229   , piece_id=TXFF2Z6H4PWWCTZFDH5YTYOO2F26M7S7U2YFGIHS6RBWS3WPEE7A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.031612082 (561h12m51.902695127s) [noise=true , address=145.79.227.50:4373   , piece_id=UDX6TREA5ZV2ZBAOIIY7EWXZR3HZ7GV6ZO5PKMXDUDD4YCJBXNCQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.031658311 (561h12m51.902741346s) [noise=true , address=145.79.227.52:4445   , piece_id=P7E6RRMQ3RMKAUBU4DPWOBQYKXUZFPUADTEG6GO7LJ6VD3QIDZ4A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.034878913 (561h12m51.905961948s) [noise=true , address=145.79.227.2:4114    , piece_id=GIDC4Z4FLBKHAQUJVMOD7GIB4GFMUVV6ITEG43RVE7M5ZH42PQ6A, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.040500507 (561h12m51.911583552s) [noise=true , address=145.79.227.11:4730   , piece_id=2JN5ANVFOYXTUMCDS3QLNSGCWHSWJFANBK3ZSPXJ235KTHIIOBSQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.069432146 (561h12m51.940515191s) [noise=true , address=145.79.227.4:4186    , piece_id=SZM2R5WN7HFHKCVTIS36XBGJ7I2GPIYLJFWBBIZR4LA4LL7ULSCA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.234775952 (561h12m52.105858997s) [noise=true , address=77.90.31.13:4629     , piece_id=YU6DEIYH4KVZE3YCTXWMV2I2LBB2LXDMORMD5IFU6SDPSPOQ3ARA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.328030110 (561h12m52.199113145s) [noise=true , address=45.91.116.12:4227    , piece_id=4U63YQR5FDDY735HXH6W26ATKH4Z6ASBFUOQS72OBFAXNJLRRVUQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.330336822 (561h12m52.201419857s) [noise=true , address=77.90.31.14:4662     , piece_id=2GTP6I75HR2KIFL35MPCV5T3GU7AXTZ3BAERZGU6CTSU2UIMZJQA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.341867358 (561h12m52.212950403s) [noise=true , address=45.91.116.7:4010     , piece_id=KWW6ZY7YFBKRVZIJBOKECHZ3QOQRL2IW2Y2CFXEW2KHAACKFA7QQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.386681009 (561h12m52.257764054s) [noise=true , address=45.91.116.10:4083    , piece_id=T5KWFMNVVGAII5NCQN4WFLO3HTK5NUASSKD2HL36T3H44LTFMI4Q, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.387083865 (561h12m52.25816691s) [noise=true , address=45.91.116.8:4046     , piece_id=UXAGDZGUC5AORPCUC3RG42KSL726ZZ6KQSIUIU3JZMPNBEPVJXQA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.952947611 (561h12m52.824030656s) [noise=true , address=77.90.31.11:4554     , piece_id=EHGKDAWWHT5XURIH22FIFAI53FUGX6WHRFELZUHA7XSS25AYNRRA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: <nil>
Dec 29 00:41:52.952971467 (561h12m52.824054522s) [seg_pos={0 0}, upload=492031]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:41:52.953201746 (561h12m52.824284781s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953209971 (561h12m52.824293016s) [noise=true , address=45.91.116.9:4263     , piece_id=W4BKLFVATF6XSPDCJK5XJKPYGZMDNMESOQDE27DMKJ5D6T4CN7ZQ, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953326773 (561h12m52.824409848s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953350198 (561h12m52.824433243s) [noise=true , address=145.79.227.35:4483   , piece_id=JVGF63CEKZNUJNN6JULVD6TWSR5FWJNAVJBFUJXYG2OUOWAB7BMA, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953474654 (561h12m52.824557699s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 29 00:41:52.953484374 (561h12m52.824567419s) [noise=true , address=45.91.116.6:4974     , piece_id=SBR3RUIH2AQVHP4OBKFTNAQPI7NAFO3WJ6QRFONHHMTYQ5WXMJVQ, seg_pos={0 0}, upload=492031]: Long tail cancellation due to optimal threshold
Dec 29 00:41:52.953505583 (561h12m52.824588628s) [seg_pos={0 0}, upload=492031]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:41:52.953611595 (561h12m52.82469464s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:41:52.953617466 (561h12m52.824700521s) [upload=492031]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:41:53.239743086 (561h12m53.110826121s) [upload=492031]: Done uploading segments...
Dec 29 00:41:53.239762553 (561h12m53.110845588s) [upload=492031]: Upload finished. err: <nil>
Duration: 30.003625239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [...........X...X........X.X........X.X..........X...........]
duration: 
30.465987273s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:41:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 30.466017129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [............X...X........X.X........X.X..........X..........]
duration: 
25.471434511s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:41:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 25.471470489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [................X...X.X...X..X..X.X..X..X.......X...........]
duration: 
5.976314853s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:40:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 5.9763388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.............X...X........X.X........X.X..........X.........]
duration: 
19.640499231s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:40:36 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 19.640528197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..............X...X........X.X........X.X..........X........]
duration: 
23.06461324s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:40:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 23.064642346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [.................X...X.X...X..X..X.X..X..X.......X..........]
duration: 
5.829409828s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:39:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 5.829437591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [...............X...X........X.X........X.X..........X.......]
duration: 
15.420009852s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:39:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 15.42004582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [..................X...X.X...X..X..X.X..X..X.......X.........]
duration: 
6.080495017s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:39:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 6.080518021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [................X...X........X.X........X.X..........X......]
duration: 
24.738492445s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:39:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 24.738527902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.................X...X........X.X........X.X..........X.....]
duration: 
15.39309101s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:38:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 15.393120074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [...................X...X.X...X..X..X.X..X..X.......X........]
duration: 
6.013054172s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:38:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 6.013075222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..................X...X........X.X........X.X..........X....]
duration: 
20.87931819s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:37:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 20.879335703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [....................X...X.X...X..X..X.X..X..X.......X.......]
duration: 
5.862337942s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:37:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 5.862365864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [...................X...X........X.X........X.X..........X...]
duration: 
22.025511451s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:36:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 22.025538033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [.....................X...X.X...X..X..X.X..X..X.......X......]
duration: 
5.993540331s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:36:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 5.993564397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [....................X...X........X.X........X.X..........X..]
duration: 
15.357420454s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:36:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 15.357447354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [......................X...X.X...X..X..X.X..X..X.......X.....]
duration: 
5.99423765s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:35:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 5.994265552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.....................X...X........X.X........X.X..........X.]
duration: 
37.758448493s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:35:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 37.758466577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [......................X...X........X.X........X.X..........X]
duration: 
1m0.007956955s
trace: 
https://jaeger.dev.storj.io/trace/4638e2a37cbcfb03
FAIL (2025-12-29 00:35:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875111976 (561h5m2.746195011s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.875831988 (561h5m2.746915033s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876337831 (561h5m2.747420866s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876431559 (561h5m2.747514604s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876771086 (561h5m2.747854131s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.876900903 (561h5m2.747983949s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877064715 (561h5m2.74814775s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877346141 (561h5m2.748429187s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877435802 (561h5m2.748518837s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.877699364 (561h5m2.748782399s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878141936 (561h5m2.749224981s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878273246 (561h5m2.749356281s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878579139 (561h5m2.749662184s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878684830 (561h5m2.749767875s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.878944153 (561h5m2.750027199s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879232092 (561h5m2.750315138s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:02.879879835 (561h5m2.75096288s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Uploading piece...
Dec 29 00:34:04.331348777 (561h5m4.202431812s) [upload=491909]: Next returned nil segment
Dec 29 00:34:04.331362342 (561h5m4.202445387s) [upload=491909]: Waiting for error group managing segments...
Dec 29 00:34:04.713173648 (561h5m4.584256683s) [noise=true , address=64.49.13.140:4505    , piece_id=CO34EQEP3QZTPTMFDTBD7Z2727I4K37KM2ZFYNCW5B75FW5W4JIQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.741162402 (561h5m4.612245437s) [noise=true , address=145.79.227.9:4393    , piece_id=NAHQIN7L4DNJ3HEN2P33NSK44VW36JECVYOZWVCSDW2KQIGSPQSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.750639197 (561h5m4.621722242s) [noise=true , address=145.79.227.18:4012   , piece_id=ETBBIRYNQGIYY5NJLLDEUBMTZMOQ7N2XK7B2CCBPFNBJ2HHOZSJQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.775357414 (561h5m4.646440459s) [noise=true , address=145.79.227.3:4165    , piece_id=IWSPEVHZVH47JPDQ6272U4774ADBXNC4BOVOJ4ZY63BQF7CH4AVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:04.967012083 (561h5m4.838095128s) [noise=true , address=5.231.73.4:4021      , piece_id=2UFVHM227I52JLBQCEA2AWBXIIKJKK5RPIFSFB4NBO6ZMTJGEWSQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.014988377 (561h5m4.886071422s) [noise=true , address=77.90.31.11:4579     , piece_id=GCE3QBJVS2VX6SQAZNRGKHM2E2FCDWFHNSUZSNWJ2HLAN5DPINHQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.026786663 (561h5m4.897869698s) [noise=true , address=77.90.31.16:4737     , piece_id=XBFJEZ2N3WO5QQYLUCAJXRAHIUHEBBMSWKHL2UIN6KOYYVMQCACQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.196951211 (561h5m5.068034256s) [noise=true , address=77.90.31.12:4624     , piece_id=VPJ4Y2E6ROHJBA4DQU764MMUQBB6DSB2FPXJ2VZD5MUOIJWCWWTA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.354372154 (561h5m5.225455189s) [noise=true , address=77.90.31.14:4671     , piece_id=F6TOWCTFB52KD36DQF35ORZSIVTDRQO44AI5UWE5IAYIWF2YVUMA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.671945690 (561h5m5.543028725s) [noise=true , address=145.79.227.30:4788   , piece_id=ZXD2DWKU6GVLIH26VGS2KOE2EDFS6WDU6DCJLWYKPALZAIVWTZQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.687586602 (561h5m5.558669648s) [noise=true , address=145.79.227.44:4223   , piece_id=KOMUEKBNZODNAEAZ7LHTBIM37OC5YTJVOEF5LCQICF4HUQ32YKQA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.704607122 (561h5m5.575690167s) [noise=true , address=145.79.227.49:4362   , piece_id=3YXZZQBST4MK3UVPVSRS6SDLIJFYA26STUVZYFR2R3VXUU3DMITA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.738853602 (561h5m5.609936647s) [noise=true , address=145.79.227.43:4177   , piece_id=AHXMAIIHSIJJWTYP53BSHPLNAH6D5SGGNMKQ3OAPWLPHSCWRWFVA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761962104 (561h5m5.633045139s) [noise=true , address=5.231.73.3:4032      , piece_id=BF2BXCPL5DJTJJQD5GZYXO67TMDFJUSACY6JLSV7DUVUVRQONLJA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: <nil>
Dec 29 00:34:05.761986280 (561h5m5.633069325s) [seg_pos={0 0}, upload=491909]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:34:05.762232539 (561h5m5.633315584s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762241796 (561h5m5.633324841s) [noise=true , address=145.79.227.40:4075   , piece_id=BMKKPOI4ASVVXGYYRZZAFLLTTZH4JEJQBI6Y3OWGRX56RRHJYBNQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762493657 (561h5m5.633576702s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762503305 (561h5m5.63358635s) [noise=true , address=77.90.31.18:4836     , piece_id=OBTLWG5ISP664KNJ7JF7LEYTHSCS556MNELJKGI42ESEK4ZVTICQ, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762669461 (561h5m5.633752506s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762677897 (561h5m5.633760942s) [noise=true , address=77.90.31.10:4549     , piece_id=HC4RBHZ552ZEEX56CNQHH2PGY66YXHIND4J7SLEVTVBQX4MZ3J4Q, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762884831 (561h5m5.633967896s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSJZtVcbbYZqZQRz96m51gH7Loe4RbXWaViejXLfhPsFoiqXi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:34:05.762894099 (561h5m5.633977144s) [noise=true , address=5.231.73.2:4019      , piece_id=4VQNBNVISNKKCUEKI47DEATWSGIOWHLV6YKPOT6O4POOGNJPY7BA, seg_pos={0 0}, upload=491909]: Long tail cancellation due to optimal threshold
Dec 29 00:34:05.762915209 (561h5m5.633998244s) [seg_pos={0 0}, upload=491909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:34:05.763078920 (561h5m5.634161965s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:34:05.763086245 (561h5m5.63416929s) [upload=491909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:34:06.049585652 (561h5m5.920668697s) [upload=491909]: Done uploading segments...
Dec 29 00:34:06.049605089 (561h5m5.920688124s) [upload=491909]: Upload finished. err: <nil>
Duration: 1m0.007983916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [.......................X...X.X...X..X..X.X..X..X.......X....]
duration: 
6.007932577s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:34:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 6.00796044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [........................X...X.X...X..X..X.X..X..X.......X...]
duration: 
5.875450396s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:34:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 5.875480504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.......................X...X........X.X........X.X..........]
duration: 
37.896418747s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:33:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 37.896462239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [.........................X...X.X...X..X..X.X..X..X.......X..]
duration: 
8.760472944s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:33:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 8.760497701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [..........................X...X.X...X..X..X.X..X..X.......X.]
duration: 
6.028768491s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:32:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 6.028785302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [........................X...X........X.X........X.X.........]
duration: 
15.447389335s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:32:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 15.447420736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [...........................X...X.X...X..X..X.X..X..X.......X]
duration: 
30.003454513s
trace: 
https://jaeger.dev.storj.io/trace/2213e1679f324a6c
FAIL (2025-12-29 00:32:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2918 (561h2m57.022875953s) [upload=491877]: Uploading segments...
Dec 29 00:31:57.152135140 (561h2m57.023218175s) [upload=491877]: Got next segment. Inline: false
Dec 29 00:31:57.152209942 (561h2m57.023292977s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:31:57.152214471 (561h2m57.023297526s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:31:57.399599298 (561h2m57.270682333s) [seg_pos={0 0}, upload=491877]: Begin upload segment...
Dec 29 00:31:57.399669090 (561h2m57.270752125s) [seg_pos={0 0}, upload=491877]: Done begin upload segment.
Dec 29 00:31:57.399702394 (561h2m57.270785429s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.400625843 (561h2m57.271708878s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401155301 (561h2m57.272238366s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.401998155 (561h2m57.273081201s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.402594050 (561h2m57.273677095s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.403577573 (561h2m57.274660608s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.404574491 (561h2m57.275657526s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.406676213 (561h2m57.277759248s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.408890910 (561h2m57.279973946s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.411015967 (561h2m57.282099012s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.412729129 (561h2m57.283812164s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.414385734 (561h2m57.285468769s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.416829096 (561h2m57.287912131s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.418546726 (561h2m57.289629761s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.420135222 (561h2m57.291218257s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:57.421642251 (561h2m57.292725286s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Uploading piece...
Dec 29 00:31:59.145209917 (561h2m59.016292962s) [upload=491877]: Next returned nil segment
Dec 29 00:31:59.145226458 (561h2m59.016309493s) [upload=491877]: Waiting for error group managing segments...
Dec 29 00:31:59.433982663 (561h2m59.305065698s) [noise=true , address=145.79.227.5:4222    , piece_id=7G545PB6P5DAHGZGWFK7BUVLMQG76MXUMAHHAUFIFDL4OWMXUOVQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.434833674 (561h2m59.305916709s) [noise=true , address=145.79.227.37:4567   , piece_id=KOUY7O6RCWTBXYP6IKEZ6RC5QDR36ZBW42CGTO3I3EJ4NP5C7NIA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.442895063 (561h2m59.313978108s) [noise=true , address=145.79.227.42:4121   , piece_id=XCIADXQX2VJ3YAX6KMAR4KSL76I3MFRITZSZ6WLJ7QTPKDQOMOYQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446608094 (561h2m59.317691139s) [noise=true , address=145.79.227.2:4115    , piece_id=SOFGV5JGTC5EROGPSZGWNHAPJMNIGBU37O72O6MCQIKXPUHLGQTA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.446761145 (561h2m59.31784418s) [noise=true , address=145.79.227.12:4765   , piece_id=X7OHDG7NAIQRKYWMGFX35JHJKHAQ5ZCZNGVD37PVFRLMXZDMOWXQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.447319829 (561h2m59.318402864s) [noise=true , address=145.79.227.26:4654   , piece_id=MX5YWJZK2STXFY47WJATTADGCI24CR7DO3KLBUHVTLKEWDZ4B46A, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.474965580 (561h2m59.346048615s) [noise=true , address=145.79.227.38:4979   , piece_id=U2TJDQHSHIZAQFHJP6LNTZKKQ2NPERYSXIELBLO5ELEZ3ZDYULYA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.478275943 (561h2m59.349358978s) [noise=true , address=145.79.227.16:4910   , piece_id=XJJIV3OJBUCMZFEKRNKAKKPXSORWRLJRQRVT3Q6W42KNHUWZQTPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.660860922 (561h2m59.531943957s) [noise=true , address=45.91.116.4:4937     , piece_id=U2WCI43LPJO7CSH6GDC3NSOAE7CWZR2YE7G4HKDBRDFGU3UFYCAQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.714701182 (561h2m59.585784217s) [noise=true , address=77.90.31.10:4519     , piece_id=HBVNAL7SFMKCAFQR2WBIDU2QVGK67NX6VQNHDOOJ6UOB3IY4MFPQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:31:59.802810706 (561h2m59.673893741s) [noise=true , address=45.91.116.8:4047     , piece_id=BYULUHL6U3I5IZSUJREDVSWXSAGOTNG7COCW2NNZVVHE4DRSHXCQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.108881460 (561h2m59.979964495s) [noise=true , address=45.91.116.9:4264     , piece_id=PBDUAVADWDFF2BPR342GSG2ATYIKWCWGSNOKC5BLGGVNK76IM4UA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183898864 (561h3m0.054981909s) [noise=true , address=45.91.116.6:4975     , piece_id=4JUKSHWIO4C2KPBFKCQJOC7UR5VE6OQUJLLNMYLGJBIOEPUUMVBA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: <nil>
Dec 29 00:32:00.183919492 (561h3m0.055002538s) [seg_pos={0 0}, upload=491877]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:32:00.184095248 (561h3m0.055178283s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56706->45.91.116.12:4228: use of closed network connection
Dec 29 00:32:00.184100939 (561h3m0.055183974s) [noise=true , address=45.91.116.12:4228    , piece_id=EQK36IXMVXZCVM66OPJ3PAO7NI4OTSERYDJCWZ7PUGGGCRHTOLXA, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184285580 (561h3m0.055368615s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 00:32:00.184302832 (561h3m0.055385867s) [noise=true , address=45.91.116.5:4298     , piece_id=BBLQNLGCL3NFBFTEJFRYHFNY42L6EKNXVYFETC2IUGEOXRZGQOMQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184322821 (561h3m0.055405856s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41694->45.91.116.11:4117: use of closed network connection
Dec 29 00:32:00.184345034 (561h3m0.055428069s) [noise=true , address=45.91.116.11:4117    , piece_id=VR3KX23GTS4736SU2XSWET3QEPQHELJIUUGTP3JPFLAGWNSOQGSQ, seg_pos={0 0}, upload=491877]: Long tail cancellation due to optimal threshold
Dec 29 00:32:00.184440755 (561h3m0.0555238s) [seg_pos={0 0}, upload=491877]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:32:00.184548000 (561h3m0.055631035s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:32:00.184553961 (561h3m0.055637006s) [upload=491877]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:32:00.470018797 (561h3m0.341101852s) [upload=491877]: Done uploading segments...
Dec 29 00:32:00.470053814 (561h3m0.341136859s) [upload=491877]: Upload finished. err: <nil>
Duration: 30.00346805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.........................X...X........X.X........X.X........]
duration: 
17.200997177s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:32:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 17.201012355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [..........................X...X........X.X........X.X.......]
duration: 
26.816575382s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:31:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 26.816603194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [...........................X...X........X.X........X.X......]
duration: 
8.293444816s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:30:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 8.293477878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [............................X...X.X...X..X..X.X..X..X.......]
duration: 
6.388475469s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:30:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 6.388503692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [............................X...X........X.X........X.X.....]
duration: 
44.479874482s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:30:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 44.479906302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [.............................X...X.X...X..X..X.X..X..X......]
duration: 
5.879297857s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:30:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 5.879329678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [X.............................X...X.X...X..X..X.X..X..X.....]
duration: 
5.944263648s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:29:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 5.944288836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.............................X...X........X.X........X.X....]
duration: 
20.69036295s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:29:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 20.690383739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [.X.............................X...X.X...X..X..X.X..X..X....]
duration: 
5.928467744s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:28:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 5.928494303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [..X.............................X...X.X...X..X..X.X..X..X...]
duration: 
5.6631123s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:28:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 5.663140352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [..............................X...X........X.X........X.X...]
duration: 
15.427413103s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:28:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 15.427426068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [...............................X...X........X.X........X.X..]
duration: 
37.456964274s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:28:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 37.456979714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [................................X...X........X.X........X.X.]
duration: 
17.163673279s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:27:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 17.163690892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [...X.............................X...X.X...X..X..X.X..X..X..]
duration: 
6.034889343s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:27:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 6.03491375s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.................................X...X........X.X........X.X]
duration: 
1m0.005056176s
trace: 
https://jaeger.dev.storj.io/trace/134e4209a37c7caf
FAIL (2025-12-29 00:27:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
26:13.191227882 (560h57m13.062310927s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191721532 (560h57m13.062804567s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.191880335 (560h57m13.06296338s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192187099 (560h57m13.063270144s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192215743 (560h57m13.063298788s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192465269 (560h57m13.063548314s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192521305 (560h57m13.06360435s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192734280 (560h57m13.063817315s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.192777513 (560h57m13.063860548s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193018481 (560h57m13.064101516s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193022650 (560h57m13.064105685s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193319415 (560h57m13.06440246s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193409376 (560h57m13.064492421s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193622432 (560h57m13.064705477s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.193832852 (560h57m13.064915898s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194046930 (560h57m13.065129965s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:13.194183640 (560h57m13.065266675s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Uploading piece...
Dec 29 00:26:14.068499486 (560h57m13.939582522s) [noise=true , address=64.49.13.26:4945     , piece_id=BH7BYFSDMBIGVLRSWKAIV5LU7NBMCVJ57XKZKABMOEISOAI5NNWQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000
Dec 29 00:26:14.484230550 (560h57m14.355313605s) [upload=491795]: Next returned nil segment
Dec 29 00:26:14.484251320 (560h57m14.355334355s) [upload=491795]: Waiting for error group managing segments...
Dec 29 00:26:14.822578834 (560h57m14.69366187s) [noise=true , address=64.49.13.140:4506    , piece_id=HCHS3QHE2TGNNZPC3HBDE7ZLOM666EC3S3RJ6325LZFBDLYWQUKQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:14.842111849 (560h57m14.713194884s) [noise=true , address=64.49.13.140:4507    , piece_id=AR2C5UEPRFADV7HLS6RYEHRMKT4IN5UDSEL4N33MVMWZV4IJSMPQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.065418778 (560h57m14.936501813s) [noise=true , address=145.79.227.38:4980   , piece_id=5DMV66J72JHWN5BJJNXPDO6MYOWXYPPSSD4U2662MX6LDPNBX4SA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.082037051 (560h57m14.953120086s) [noise=true , address=145.79.227.29:4963   , piece_id=63XC33VN6HJYGVTJK64MHXROSB3IXF7C73JPXYFM5Y3UD6VIEB6A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.099790536 (560h57m14.970873581s) [noise=true , address=145.79.227.32:4859   , piece_id=Y7QXAESCWRZCOFTDZ6QOFMKJZJ2PSPK66272A2MDT5AEF7PE6KUQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.119655683 (560h57m14.990738718s) [noise=true , address=5.231.73.2:4018      , piece_id=IMK54FLBPOGG5WA55NU4ZAEAVRPJAUI5ZMEF6GSZHP5STMKKMWEA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.176980828 (560h57m15.048063863s) [noise=true , address=5.231.73.5:4035      , piece_id=RN44ASIEXLQB72UTPX7MX2KLLCOLMHXL2ETYGZLJFLLBSIYODPBQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.247798214 (560h57m15.11888126s) [noise=true , address=77.90.31.12:4602     , piece_id=Y6HXD2BEVIOWWNYQVQXUVB3XLQKN3BMCPUIPWIAN2GKX6IICARTA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.299200442 (560h57m15.170283477s) [noise=true , address=45.91.116.13:4180    , piece_id=GS25ANWTX73JOFSCZ5W5KBP3O4BL5EN6PT2VFHQHIK3D426AYFAA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.345385986 (560h57m15.216469021s) [noise=true , address=145.79.227.14:4852   , piece_id=2X2GV3RKY4CAGMVPPZCVU6XAFRQFCZVZRBZ63YG6CNWKDQ6ROTLQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.399083895 (560h57m15.27016693s) [noise=true , address=77.90.31.18:4830     , piece_id=QPOIIU2HQVMSA2UQE6WLPNEFCN3U6MHSROOKTPJZWQ6NEVDOD7KA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.920938724 (560h57m15.792021779s) [noise=true , address=145.79.227.40:4069   , piece_id=FHXDPSRFLOHWNT2ZPQGCLAVJPMU5SUFLTM6JOWBLHBS3YKN6MWCQ, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:15.968383506 (560h57m15.839466541s) [noise=true , address=145.79.227.35:4505   , piece_id=QQM47F47HEKXLTDMKTOPCDSW7RKJ54HONZLNFBECGBWST5YJCF7Q, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007566315 (560h57m15.878649401s) [noise=true , address=145.79.227.34:4928   , piece_id=4XRPAVN5BKYBEBBPHDRU7UVZV6T4HGDUY4FGMTWODSI4EVRJBUYA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: <nil>
Dec 29 00:26:16.007593978 (560h57m15.878677023s) [seg_pos={0 0}, upload=491795]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:26:16.007814067 (560h57m15.878897112s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg): protocol: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33868->145.79.227.7:4300: use of closed network connection
Dec 29 00:26:16.007822614 (560h57m15.878905659s) [noise=true , address=145.79.227.7:4300    , piece_id=IAO2LG2KF4KLWJFPP6ZO4RIQSPIC5RNCQVMI3WXD5NQXVOWUQP4A, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008167411 (560h57m15.879250446s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008176117 (560h57m15.879259152s) [noise=true , address=145.79.227.49:4371   , piece_id=IWMRS6TTBWH57PMXCN5IZ7WBBD4RSWJYWK45BVDJ7X3HGPYVJFXA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008326934 (560h57m15.879409979s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:16.008334658 (560h57m15.879417703s) [noise=true , address=145.79.227.33:4887   , piece_id=XCY5447REYQKBO7DYTXA35PTBZ26JRHM2XLVCH66JO3AKSQO4AKA, seg_pos={0 0}, upload=491795]: Long tail cancellation due to optimal threshold
Dec 29 00:26:16.008355298 (560h57m15.879438343s) [seg_pos={0 0}, upload=491795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:26:16.008436532 (560h57m15.879519578s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:16.008443907 (560h57m15.879526942s) [upload=491795]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:16.303505477 (560h57m16.174588522s) [upload=491795]: Done uploading segments...
Dec 29 00:26:16.303523753 (560h57m16.174606788s) [upload=491795]: Upload finished. err: <nil>
Duration: 1m0.00510592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [X...X.............................X...X.X...X..X..X.X..X..X.]
duration: 
6.04123979s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:26:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 6.041255809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
11 out of 60 is failed (threshold=3) [.X...X.............................X...X.X...X..X..X.X..X..X]
duration: 
30.002941825s
trace: 
https://jaeger.dev.storj.io/trace/6cbb425dd7ca8af2
FAIL (2025-12-29 00:26:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:26:06.570678231 (560h57m6.441761276s) [upload=491794]: Starting upload
Dec 29 00:26:06.570766780 (560h57m6.441849825s) [upload=491794]: Done starting upload
Dec 29 00:26:06.570826854 (560h57m6.441909889s) [upload=491794]: Uploading segments...
Dec 29 00:26:06.571020613 (560h57m6.442103668s) [upload=491794]: Got next segment. Inline: false
Dec 29 00:26:06.571070328 (560h57m6.442153373s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:26:06.571076049 (560h57m6.442159084s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:26:06.815468407 (560h57m6.686551442s) [seg_pos={0 0}, upload=491794]: Begin upload segment...
Dec 29 00:26:06.815520026 (560h57m6.686603351s) [seg_pos={0 0}, upload=491794]: Done begin upload segment.
Dec 29 00:26:06.815577244 (560h57m6.686660279s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.815590169 (560h57m6.686673204s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816147400 (560h57m6.687230445s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816569634 (560h57m6.687652669s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.816921574 (560h57m6.688004619s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.817877284 (560h57m6.688960319s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818235697 (560h57m6.689318742s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.818548653 (560h57m6.689631698s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.819124288 (560h57m6.690207323s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.820204255 (560h57m6.69128729s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.821209689 (560h57m6.692292724s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.822299095 (560h57m6.69338214s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.823451890 (560h57m6.694534925s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.825206300 (560h57m6.696289335s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.826559757 (560h57m6.697642792s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:06.827832099 (560h57m6.698915144s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Uploading piece...
Dec 29 00:26:08.533843877 (560h57m8.404926912s) [upload=491794]: Next returned nil segment
Dec 29 00:26:08.533866510 (560h57m8.404949545s) [upload=491794]: Waiting for error group managing segments...
Dec 29 00:26:08.820495995 (560h57m8.69157903s) [noise=true , address=145.79.227.48:4302   , piece_id=HAQ2FG4UJHZEKLIW66INXPPSIJGTTM5ZHW2ZUCFNYE2YGDL7P3PA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.821398533 (560h57m8.692481568s) [noise=true , address=145.79.227.18:4006   , piece_id=RCTSBFLNDBSIGPQUWIVJN2VNC4NPQBM7JVEIDUO4EHMGIYI7K5MA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.827388980 (560h57m8.698472025s) [noise=true , address=145.79.227.15:4876   , piece_id=W7SMJKBK6BI4FET7DDNQT6XEVZQGVCMR7JAMHBMNMQ3YZF7PGMNA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.828657756 (560h57m8.699740791s) [noise=true , address=145.79.227.26:4654   , piece_id=NQXEE652OX67YAQ5C7NVW5NYKD3HS3RHPWCKGLL44M6FJYXPDRBQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.837657683 (560h57m8.708740718s) [noise=true , address=145.79.227.51:4410   , piece_id=OTUAHMP2MVZJFTTLHDK55MAULN22GXZPYUQNPWW6SXWHBACO6AQQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.838285238 (560h57m8.709368273s) [noise=true , address=145.79.227.19:4078   , piece_id=VDAZZ24LWRZT5722BO3JGVXSCTKOVNDFHTYHSSYSNJWGLZGDOX4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.840961674 (560h57m8.712044709s) [noise=true , address=145.79.227.16:4909   , piece_id=JOPEOYABKLRNWFSKTGZKIL34TNDY4NDZMHQ42E67TKHYUIJE7F4Q, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.844951822 (560h57m8.716034867s) [noise=true , address=145.79.227.39:4014   , piece_id=PDSQAH7NTI4F5AK64ZGVHYMDSG4P7CWNIF7I2YQTXMSOYE3V6GTQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.845488334 (560h57m8.716571369s) [noise=true , address=145.79.227.12:4765   , piece_id=C3EOQ5PUAKT4GNKUNNV7JDJHS3XGRYDTUX7W54W6SUEGN4HWKR2A, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:08.846019304 (560h57m8.71710234s) [noise=true , address=145.79.227.5:4222    , piece_id=NWXXEODEMUMZ5KXGMCL2EFAPQTUOB5ARR3CKO37EHT2UB6L5TUKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.063311892 (560h57m8.934394947s) [noise=true , address=45.91.116.8:4045     , piece_id=PCQCFEVFWUV65KWBWS4QKRWUXBGMX4UYEKF47LBNO2F6CJVGSSPQ, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.066490083 (560h57m8.937573128s) [noise=true , address=45.91.116.12:4225    , piece_id=MNJWZPKD5EKXJ5UBRFW2DL43E6O3MHR3DALSMZQCRYV3ISMK57SA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074598844 (560h57m8.945681889s) [noise=true , address=45.91.116.10:4083    , piece_id=VECVEUVQS4CPC7VNOGJPU4O5IGF4S3IC2LY6FG6Y5CXWVULLGVSA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: <nil>
Dec 29 00:26:09.074623631 (560h57m8.945706676s) [seg_pos={0 0}, upload=491794]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:26:09.074856123 (560h57m8.945939168s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49260->145.79.227.2:4115: use of closed network connection
Dec 29 00:26:09.074864841 (560h57m8.945947886s) [noise=true , address=145.79.227.2:4115    , piece_id=DUWJJKJKW5B2Z4ACNJSTAQ5LIZBOUNCBBRVZXLJWTYJ6FTKNRUBA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075220638 (560h57m8.946303683s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075230396 (560h57m8.946313431s) [noise=true , address=145.79.227.44:4194   , piece_id=MUDEM3GR7G5GZSV3W5QZS5VAH2OZU7V6O3KFW7C7KLPBVMUIRSQA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075404538 (560h57m8.946487583s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 29 00:26:09.075413615 (560h57m8.94649666s) [noise=true , address=45.91.116.4:4940     , piece_id=NZUABIST4M33QQQI7RNDMI26RJXT7FSDEUNF6C4RTHBHANNFEWKA, seg_pos={0 0}, upload=491794]: Long tail cancellation due to optimal threshold
Dec 29 00:26:09.075435026 (560h57m8.946518071s) [seg_pos={0 0}, upload=491794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:26:09.075520879 (560h57m8.946603924s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:26:09.075528534 (560h57m8.94661158s) [upload=491794]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:26:09.358282119 (560h57m9.229365184s) [upload=491794]: Done uploading segments...
Dec 29 00:26:09.358408248 (560h57m9.229491283s) [upload=491794]: Upload finished. err: <nil>
Duration: 30.002966483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [..................................X...X........X.X........X.]
duration: 
35.869389759s
trace: 
https://jaeger.dev.storj.io/trace/2b0d256f8d515e28
FAIL (2025-12-29 00:26:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rNDTtTWVFcGgWM4ZWJ4GKhoATGLJTeSUYnBDQKjeRasQ3U8cx, Piece ID: LNP5MXJEIPO34222QAAPCRLA3SYG2NKWWWRDARY7HXQK6FHUGU2A) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: TWEP4CCIS6MHUTNORU3JG3KDK4THLETLUUPDNV73DYYE2MWMW5KQ) context deadline exceeded; piecestore: (Node ID: 12uCbmPHVXpukjQahs2G6oh7VY2zDMooMAbdgSyHN7Biuesmz3A, Piece ID: GTLIEGKADHBKI6KB4MVJZ6NOQFEP3YSPMSIO3BNNBWD5ZHC2ZUCQ) context deadline exceeded
======LOGS======
4:29.285158647 (560h55m29.156241682s) [noise=true , address=145.79.227.9:4397    , piece_id=CBJX36UGVBM2YP5YIXHRPOTISBUPUVTG7FTPZ5DNZLNNADEKMEAA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.285609425 (560h55m29.15669246s) [noise=true , address=145.79.227.40:4065   , piece_id=WM3VFWHLWHMNRVUSIIROHYN3JQWESPGU6XLRBR5DQZ7D6FRKKXZA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.285669861 (560h55m29.156752896s) [noise=true , address=5.231.73.5:4031      , piece_id=LNP5MXJEIPO34222QAAPCRLA3SYG2NKWWWRDARY7HXQK6FHUGU2A, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.286021350 (560h55m29.157104395s) [noise=true , address=77.90.31.19:4866     , piece_id=GTLIEGKADHBKI6KB4MVJZ6NOQFEP3YSPMSIO3BNNBWD5ZHC2ZUCQ, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.286146318 (560h55m29.157229353s) [noise=true , address=193.149.164.143:4703 , piece_id=WVU7H3LEBZLB74WWWIIWIQU4SVSR7BDQ64HY3CJPSKKOMNKTIBCA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.286342621 (560h55m29.157425656s) [noise=true , address=77.90.31.15:4703     , piece_id=V4P2NGFF44EVXDHWYWRF6FLRLOLHQKTVHPLLOEQVKAXRVNUM74NA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.286459044 (560h55m29.157542079s) [noise=true , address=45.91.116.12:4239    , piece_id=3FNW77NRL3ZARKS4T6XY3TAR5QAAOHFSZY7KTUKUKWJI23GPCK6Q, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.286602457 (560h55m29.157685502s) [noise=true , address=77.90.31.18:4820     , piece_id=KEFWG2ANX7LACPUHGKACNV6KEIDXKQHTBOPMDXYOXZG5XJ2ZFFPQ, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.286760076 (560h55m29.157843111s) [noise=true , address=145.79.227.30:4777   , piece_id=B4D7XCSPMIHYFUUPI2WO562HCYOEHB3RGKS7AS7WERDQR34JSYDA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.286891226 (560h55m29.157974271s) [noise=true , address=77.90.31.14:4695     , piece_id=W4BJHHZDSAKQM5M44QP4AMWAQ73QJFYRLPR3JK3DDCQ7VU6XPO4Q, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.287060238 (560h55m29.158143273s) [noise=true , address=45.91.116.5:4314     , piece_id=TWEP4CCIS6MHUTNORU3JG3KDK4THLETLUUPDNV73DYYE2MWMW5KQ, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.287194754 (560h55m29.158277789s) [noise=true , address=145.79.227.35:4509   , piece_id=CL5BP7U4CIYE5RGQCQVVWNFV443DQV3IMPYOKRM7N75FNU7SGGSA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.287324552 (560h55m29.158407597s) [noise=true , address=145.79.227.43:4187   , piece_id=SHUSSJMTBABNHT76LZ56TU5WFOZFV4M4P62RHSHKPOISQ7OZOL3A, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.287454749 (560h55m29.158537784s) [noise=true , address=77.90.31.21:4881     , piece_id=W4BRJ4BLNQ3GRTHJ55GTNZMQG7PWYSIE7C6YWF46MAEIQQ3P4ACA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.287677133 (560h55m29.158760168s) [noise=true , address=145.79.227.34:4933   , piece_id=FDZQ2ZACVQ7WW5L2Z5LPNL44TEITTHNSOTSMXJ6VDK3UCIJY2AOA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.287752416 (560h55m29.158835451s) [noise=true , address=77.90.31.13:4636     , piece_id=2XCGVMJC56LSZBGH5HZZ6M3BIHHQROSXROOW3CT2JLODMKCM2MMA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:29.287994267 (560h55m29.159077302s) [noise=true , address=45.91.116.9:4283     , piece_id=ER4L2NIZP6XV7W2PJJYNOWAOAGRC5FNLCE3XIHFIPEXCYKNS5VDA, seg_pos={0 0}, upload=491772]: Uploading piece...
Dec 29 00:24:30.738181526 (560h55m30.609264561s) [upload=491772]: Next returned nil segment
Dec 29 00:24:30.738205281 (560h55m30.609288316s) [upload=491772]: Waiting for error group managing segments...
Dec 29 00:24:31.161093461 (560h55m31.032176526s) [noise=true , address=145.79.227.9:4397    , piece_id=CBJX36UGVBM2YP5YIXHRPOTISBUPUVTG7FTPZ5DNZLNNADEKMEAA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.218884815 (560h55m31.08996786s) [noise=true , address=5.231.73.5:4031      , piece_id=LNP5MXJEIPO34222QAAPCRLA3SYG2NKWWWRDARY7HXQK6FHUGU2A, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.393343529 (560h55m31.264426565s) [noise=true , address=77.90.31.13:4636     , piece_id=2XCGVMJC56LSZBGH5HZZ6M3BIHHQROSXROOW3CT2JLODMKCM2MMA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.400854080 (560h55m31.271937125s) [noise=true , address=77.90.31.15:4703     , piece_id=V4P2NGFF44EVXDHWYWRF6FLRLOLHQKTVHPLLOEQVKAXRVNUM74NA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.401769062 (560h55m31.272852097s) [noise=true , address=77.90.31.18:4820     , piece_id=KEFWG2ANX7LACPUHGKACNV6KEIDXKQHTBOPMDXYOXZG5XJ2ZFFPQ, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.427403905 (560h55m31.29848694s) [noise=true , address=77.90.31.21:4881     , piece_id=W4BRJ4BLNQ3GRTHJ55GTNZMQG7PWYSIE7C6YWF46MAEIQQ3P4ACA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.476596805 (560h55m31.34767984s) [noise=true , address=45.91.116.5:4314     , piece_id=TWEP4CCIS6MHUTNORU3JG3KDK4THLETLUUPDNV73DYYE2MWMW5KQ, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.512812295 (560h55m31.38389534s) [noise=true , address=77.90.31.14:4695     , piece_id=W4BJHHZDSAKQM5M44QP4AMWAQ73QJFYRLPR3JK3DDCQ7VU6XPO4Q, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.614401696 (560h55m31.485484741s) [noise=true , address=45.91.116.12:4239    , piece_id=3FNW77NRL3ZARKS4T6XY3TAR5QAAOHFSZY7KTUKUKWJI23GPCK6Q, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:31.756681306 (560h55m31.627764341s) [noise=true , address=77.90.31.19:4866     , piece_id=GTLIEGKADHBKI6KB4MVJZ6NOQFEP3YSPMSIO3BNNBWD5ZHC2ZUCQ, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:32.042534893 (560h55m31.913617928s) [noise=true , address=145.79.227.43:4187   , piece_id=SHUSSJMTBABNHT76LZ56TU5WFOZFV4M4P62RHSHKPOISQ7OZOL3A, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:32.068628410 (560h55m31.939711445s) [noise=true , address=145.79.227.40:4065   , piece_id=WM3VFWHLWHMNRVUSIIROHYN3JQWESPGU6XLRBR5DQZ7D6FRKKXZA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:32.152272350 (560h55m32.023355386s) [noise=true , address=145.79.227.35:4509   , piece_id=CL5BP7U4CIYE5RGQCQVVWNFV443DQV3IMPYOKRM7N75FNU7SGGSA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:32.183611374 (560h55m32.054694409s) [noise=true , address=145.79.227.30:4777   , piece_id=B4D7XCSPMIHYFUUPI2WO562HCYOEHB3RGKS7AS7WERDQR34JSYDA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: <nil>
Dec 29 00:24:32.183629840 (560h55m32.054712885s) [seg_pos={0 0}, upload=491772]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:24:32.183813639 (560h55m32.054896674s) [noise=true , address=193.149.164.143:4703 , piece_id=WVU7H3LEBZLB74WWWIIWIQU4SVSR7BDQ64HY3CJPSKKOMNKTIBCA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:32.183819280 (560h55m32.054902315s) [noise=true , address=193.149.164.143:4703 , piece_id=WVU7H3LEBZLB74WWWIIWIQU4SVSR7BDQ64HY3CJPSKKOMNKTIBCA, seg_pos={0 0}, upload=491772]: Long tail cancellation due to optimal threshold
Dec 29 00:24:32.183833528 (560h55m32.054916563s) [noise=true , address=145.79.227.33:4892   , piece_id=VXLB43LT7U7VAXIBVRW44OA2KG7Z4LLSWEV77VNNRQ5HCVRIPXWQ, seg_pos={0 0}, upload=491772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:32.183841693 (560h55m32.054924738s) [noise=true , address=145.79.227.33:4892   , piece_id=VXLB43LT7U7VAXIBVRW44OA2KG7Z4LLSWEV77VNNRQ5HCVRIPXWQ, seg_pos={0 0}, upload=491772]: Long tail cancellation due to optimal threshold
Dec 29 00:24:32.183987641 (560h55m32.055070676s) [noise=true , address=45.91.116.9:4283     , piece_id=ER4L2NIZP6XV7W2PJJYNOWAOAGRC5FNLCE3XIHFIPEXCYKNS5VDA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:32.183993001 (560h55m32.055076046s) [noise=true , address=45.91.116.9:4283     , piece_id=ER4L2NIZP6XV7W2PJJYNOWAOAGRC5FNLCE3XIHFIPEXCYKNS5VDA, seg_pos={0 0}, upload=491772]: Long tail cancellation due to optimal threshold
Dec 29 00:24:32.183999403 (560h55m32.055082438s) [noise=true , address=145.79.227.34:4933   , piece_id=FDZQ2ZACVQ7WW5L2Z5LPNL44TEITTHNSOTSMXJ6VDK3UCIJY2AOA, seg_pos={0 0}, upload=491772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:32.184005234 (560h55m32.055088269s) [noise=true , address=145.79.227.34:4933   , piece_id=FDZQ2ZACVQ7WW5L2Z5LPNL44TEITTHNSOTSMXJ6VDK3UCIJY2AOA, seg_pos={0 0}, upload=491772]: Long tail cancellation due to optimal threshold
Dec 29 00:24:32.184021395 (560h55m32.05510443s) [seg_pos={0 0}, upload=491772]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:24:32.184080398 (560h55m32.055163433s) [upload=491772]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:24:32.184086309 (560h55m32.055169344s) [upload=491772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:24:32.467103295 (560h55m32.33818633s) [upload=491772]: Done uploading segments...
Dec 29 00:24:32.467122050 (560h55m32.338205085s) [upload=491772]: Upload finished. err: <nil>
Duration: 35.869404237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [..X...X.............................X...X.X...X..X..X.X..X..]
duration: 
5.929744776s
trace: 
https://jaeger.dev.storj.io/trace/50383e94a2ef611d
FAIL (2025-12-29 00:25:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ) context deadline exceeded
======LOGS======
Dec 29 00:24:42.949490598 (560h55m42.820573633s) [upload=491774]: Starting upload
Dec 29 00:24:42.949553929 (560h55m42.820636984s) [upload=491774]: Done starting upload
Dec 29 00:24:42.949587483 (560h55m42.820670518s) [upload=491774]: Uploading segments...
Dec 29 00:24:42.949778466 (560h55m42.820861501s) [upload=491774]: Got next segment. Inline: false
Dec 29 00:24:42.949824414 (560h55m42.820907449s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:24:42.949830736 (560h55m42.820913771s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:24:43.194456640 (560h55m43.065539675s) [seg_pos={0 0}, upload=491774]: Begin upload segment...
Dec 29 00:24:43.194510333 (560h55m43.065593378s) [seg_pos={0 0}, upload=491774]: Done begin upload segment.
Dec 29 00:24:43.194562321 (560h55m43.065645366s) [noise=true , address=45.91.116.14:4191    , piece_id=TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.194574224 (560h55m43.065657259s) [noise=true , address=145.79.227.14:4840   , piece_id=YKF57JEPRBSKTDHS5X74MDIZO52ZOCAUQNE4VRKPM53ZEZL4ZU2Q, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195248268 (560h55m43.066331303s) [noise=true , address=77.90.31.18:4808     , piece_id=NSOYXKG5OVP3XGU6QF6W7FYJS2MCOWKKUVW4K4BSSAL7IETWL6HQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195553659 (560h55m43.066636694s) [noise=true , address=145.79.227.51:4411   , piece_id=VS3M6XELAMEA7A23VW53FKB2FHHI3POOHH4RJ42X6HGW7NHILN6Q, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195571603 (560h55m43.066654648s) [noise=true , address=145.79.227.32:4836   , piece_id=3K4RGROKYQJRSBKJBVEVSMK7FEUXSKTDYA5QQAKXP6IEMWQX3EWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196044564 (560h55m43.067127599s) [noise=true , address=145.79.227.47:4266   , piece_id=E3JGDN7P7V2LG2MYY37DK45Q66MH5THDADG4DCS533M543KCWKGQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196475574 (560h55m43.067558609s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196829118 (560h55m43.067912153s) [noise=true , address=45.91.116.11:4120    , piece_id=64MKKHBIPSRYDM6CDSP6V4EQVMO7APNUIRCTGZCDYVHVCSLBYRRQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.197226635 (560h55m43.06830967s) [noise=true , address=145.79.227.7:4293    , piece_id=7PXD46G6X4CXV7LOX7PLJPFOCT4BWHQKQ3LP6PJRUASYKUDNK3XQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.197611127 (560h55m43.068694162s) [noise=true , address=145.79.227.39:4015   , piece_id=HXS4M7DDYMVC5KFDFILGL4JVV36O22HC6DNOMKFBSRY2HGZTADBQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199252673 (560h55m43.070335718s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199300494 (560h55m43.070383529s) [noise=true , address=145.79.227.5:4222    , piece_id=NW3TMGBBHGESSBMJ42AC2R7TNBV6VJPNYQNX4LXB4B6XZOQC7MGA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199326343 (560h55m43.070409388s) [noise=true , address=145.79.227.15:4876   , piece_id=FW7RTIIEIDUPO7U4VHILNAUXSEQUQSH3ISVCZZVI3F43PHIWMVUA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.208763172 (560h55m43.079846207s) [noise=true , address=45.91.116.10:4081    , piece_id=HVMUAUQUZ6T6EQVXAP7ZELGZDAV6L62QKMA7WIUERWKRIATRHTLA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.213399089 (560h55m43.084482135s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.218974406 (560h55m43.090057451s) [noise=true , address=45.91.116.13:4156    , piece_id=BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:44.909291255 (560h55m44.780374291s) [upload=491774]: Next returned nil segment
Dec 29 00:24:44.909311463 (560h55m44.780394498s) [upload=491774]: Waiting for error group managing segments...
Dec 29 00:24:45.200338041 (560h55m45.071421076s) [noise=true , address=145.79.227.7:4293    , piece_id=7PXD46G6X4CXV7LOX7PLJPFOCT4BWHQKQ3LP6PJRUASYKUDNK3XQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.205843164 (560h55m45.076926199s) [noise=true , address=145.79.227.5:4222    , piece_id=NW3TMGBBHGESSBMJ42AC2R7TNBV6VJPNYQNX4LXB4B6XZOQC7MGA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.211214101 (560h55m45.082297136s) [noise=true , address=145.79.227.39:4015   , piece_id=HXS4M7DDYMVC5KFDFILGL4JVV36O22HC6DNOMKFBSRY2HGZTADBQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.220374954 (560h55m45.091457989s) [noise=true , address=145.79.227.51:4411   , piece_id=VS3M6XELAMEA7A23VW53FKB2FHHI3POOHH4RJ42X6HGW7NHILN6Q, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.224941009 (560h55m45.096024054s) [noise=true , address=145.79.227.47:4266   , piece_id=E3JGDN7P7V2LG2MYY37DK45Q66MH5THDADG4DCS533M543KCWKGQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.228064747 (560h55m45.099147782s) [noise=true , address=145.79.227.32:4836   , piece_id=3K4RGROKYQJRSBKJBVEVSMK7FEUXSKTDYA5QQAKXP6IEMWQX3EWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.232510042 (560h55m45.103593077s) [noise=true , address=145.79.227.14:4840   , piece_id=YKF57JEPRBSKTDHS5X74MDIZO52ZOCAUQNE4VRKPM53ZEZL4ZU2Q, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.242384935 (560h55m45.11346797s) [noise=true , address=145.79.227.15:4876   , piece_id=FW7RTIIEIDUPO7U4VHILNAUXSEQUQSH3ISVCZZVI3F43PHIWMVUA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.439676608 (560h55m45.310759643s) [noise=true , address=45.91.116.14:4191    , piece_id=TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.480174381 (560h55m45.351257416s) [noise=true , address=77.90.31.18:4808     , piece_id=NSOYXKG5OVP3XGU6QF6W7FYJS2MCOWKKUVW4K4BSSAL7IETWL6HQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.536506666 (560h55m45.407589701s) [noise=true , address=45.91.116.11:4120    , piece_id=64MKKHBIPSRYDM6CDSP6V4EQVMO7APNUIRCTGZCDYVHVCSLBYRRQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.539246563 (560h55m45.410329618s) [noise=true , address=45.91.116.13:4156    , piece_id=BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.549986663 (560h55m45.421069708s) [noise=true , address=45.91.116.10:4081    , piece_id=HVMUAUQUZ6T6EQVXAP7ZELGZDAV6L62QKMA7WIUERWKRIATRHTLA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.550009838 (560h55m45.421092883s) [seg_pos={0 0}, upload=491774]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:24:45.550268289 (560h55m45.421351354s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550279451 (560h55m45.421362506s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550341218 (560h55m45.421424263s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550358391 (560h55m45.421441476s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550454474 (560h55m45.42153753s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550462089 (560h55m45.421545134s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550482527 (560h55m45.421565562s) [seg_pos={0 0}, upload=491774]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:24:45.550566908 (560h55m45.421649943s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:24:45.550574102 (560h55m45.421657147s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:24:45.847821875 (560h55m45.71890491s) [upload=491774]: Done uploading segments...
Dec 29 00:24:45.847845210 (560h55m45.718928255s) [upload=491774]: Upload finished. err: <nil>
Duration: 5.929761778s core uplink us-select-1