Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 44.951329909s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:10:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 44.951364075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.......XXXXXX.......XX..X.X.........XX.X..X......XX.....X..] duration: 7.224099847s trace: https://jaeger.dev.storj.io/trace/22b3ade1c92d50df FAIL (2025-12-31 20:10:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 31 20:08:45.626736104 (628h39m45.497819149s) [upload=550880]: Starting upload Dec 31 20:08:45.626804665 (628h39m45.49788771s) [upload=550880]: Done starting upload Dec 31 20:08:45.627061393 (628h39m45.498144438s) [upload=550880]: Uploading segments... Dec 31 20:08:45.627250494 (628h39m45.498333559s) [upload=550880]: Got next segment. Inline: false Dec 31 20:08:45.627323052 (628h39m45.498406107s) [upload=550880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:08:45.627351256 (628h39m45.498434321s) [upload=550880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:08:45.873963948 (628h39m45.745046983s) [seg_pos={0 0}, upload=550880]: Begin upload segment... Dec 31 20:08:45.874006550 (628h39m45.745089585s) [seg_pos={0 0}, upload=550880]: Done begin upload segment. Dec 31 20:08:45.874030585 (628h39m45.74511362s) [noise=true , address=145.79.227.4:4186 , piece_id=YH52AJGA3ETRW4FUO7SPFT7H2XKTDF5BO36DARZZHIBJ3PRFSC6Q, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874059239 (628h39m45.745142274s) [noise=true , address=145.79.227.15:4874 , piece_id=ELPCBAXMTDB32TABI36AN7K4RUIP6BVJW7EX6NSI62TN77RHSSMA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874417953 (628h39m45.745500988s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874685572 (628h39m45.745768607s) [noise=true , address=145.79.227.45:4231 , piece_id=63NGKSNJKGAILN6ECPJ7D45C7I7HAREF4WEGFQJAI42RNWZ5EU7Q, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874994280 (628h39m45.746077315s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875380014 (628h39m45.746463059s) [noise=true , address=145.79.227.20:4438 , piece_id=T6OG3TGALBDRD67QYTQJA2WOM7TFAXMQ6PU6QNF27YJV6UYX7Z2A, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875708851 (628h39m45.746791886s) [noise=true , address=45.91.116.8:4046 , piece_id=63TKYBTHKGRTGZT3BS5VEKOCRB4EV3BFTY22GUE7TWRYGJNS2ZXQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875976731 (628h39m45.747059766s) [noise=true , address=145.79.227.29:4941 , piece_id=QN6D7CPS2J6CGMOPLVNE7YCDYB5ALUJD2WRJYVKCWQKX7AZD72HQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.876333430 (628h39m45.747416466s) [noise=true , address=145.79.227.34:4906 , piece_id=XY3TADM333BZ5HJH3XR6SG63AS54JYRIM4CMBFZRI6YQOQBUJLZA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.877685394 (628h39m45.748768429s) [noise=true , address=77.90.31.21:4880 , piece_id=KLFJG2YW2MQXII6RDBKDPVXFX4BNQMMJ7SSMBG5OFTUFPCFY4TDA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.877986307 (628h39m45.749069342s) [noise=true , address=145.79.227.48:4302 , piece_id=43BZYTZ6SCK22662YBAAPNETKJVMBXISI6VRKZHZJQA5VODD57SA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.879698287 (628h39m45.750781332s) [noise=true , address=145.79.227.3:4151 , piece_id=V2FU7IRTRBMPGMWFSNCZLNSQW5UIKCSDOVBFPCQ7PG2HK6UUHVTA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.880797090 (628h39m45.751880125s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.881937301 (628h39m45.753020336s) [noise=true , address=145.79.227.25:4619 , piece_id=PJLHQRFUQS3XK2432Z7HVCB2FM5XARRF3EUANJWUBL56KOOX36DA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.883129030 (628h39m45.754212075s) [noise=true , address=45.91.116.12:4225 , piece_id=45AQV5XOE65GJNRXVLAUC4U6RVFEX674RVK46UNPEGVMO3K44BKQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.884236879 (628h39m45.755319914s) [noise=true , address=145.79.227.17:4972 , piece_id=ILGIZC2DKXFCOFRR5RVRVUNQNL5ASA2WKARMIJINQFF4LLRTVTJA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:47.580126643 (628h39m47.451209688s) [upload=550880]: Next returned nil segment Dec 31 20:08:47.580142973 (628h39m47.451226018s) [upload=550880]: Waiting for error group managing segments... Dec 31 20:08:47.866248295 (628h39m47.73733134s) [noise=true , address=145.79.227.25:4619 , piece_id=PJLHQRFUQS3XK2432Z7HVCB2FM5XARRF3EUANJWUBL56KOOX36DA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.869725967 (628h39m47.740809013s) [noise=true , address=145.79.227.15:4874 , piece_id=ELPCBAXMTDB32TABI36AN7K4RUIP6BVJW7EX6NSI62TN77RHSSMA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.870460967 (628h39m47.741544002s) [noise=true , address=145.79.227.3:4151 , piece_id=V2FU7IRTRBMPGMWFSNCZLNSQW5UIKCSDOVBFPCQ7PG2HK6UUHVTA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.871616026 (628h39m47.742699071s) [noise=true , address=145.79.227.45:4231 , piece_id=63NGKSNJKGAILN6ECPJ7D45C7I7HAREF4WEGFQJAI42RNWZ5EU7Q, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.871892693 (628h39m47.742975758s) [noise=true , address=145.79.227.20:4438 , piece_id=T6OG3TGALBDRD67QYTQJA2WOM7TFAXMQ6PU6QNF27YJV6UYX7Z2A, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.887991537 (628h39m47.759074572s) [noise=true , address=145.79.227.4:4186 , piece_id=YH52AJGA3ETRW4FUO7SPFT7H2XKTDF5BO36DARZZHIBJ3PRFSC6Q, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.898635435 (628h39m47.76971848s) [noise=true , address=145.79.227.17:4972 , piece_id=ILGIZC2DKXFCOFRR5RVRVUNQNL5ASA2WKARMIJINQFF4LLRTVTJA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.922822599 (628h39m47.793905634s) [noise=true , address=145.79.227.29:4941 , piece_id=QN6D7CPS2J6CGMOPLVNE7YCDYB5ALUJD2WRJYVKCWQKX7AZD72HQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.950286805 (628h39m47.82136984s) [noise=true , address=145.79.227.48:4302 , piece_id=43BZYTZ6SCK22662YBAAPNETKJVMBXISI6VRKZHZJQA5VODD57SA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.058506780 (628h39m47.929589845s) [noise=true , address=77.90.31.21:4880 , piece_id=KLFJG2YW2MQXII6RDBKDPVXFX4BNQMMJ7SSMBG5OFTUFPCFY4TDA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.176175496 (628h39m48.047258531s) [noise=true , address=45.91.116.8:4046 , piece_id=63TKYBTHKGRTGZT3BS5VEKOCRB4EV3BFTY22GUE7TWRYGJNS2ZXQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.295916799 (628h39m48.166999844s) [noise=true , address=145.79.227.34:4906 , piece_id=XY3TADM333BZ5HJH3XR6SG63AS54JYRIM4CMBFZRI6YQOQBUJLZA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.358484216 (628h39m48.229567261s) [noise=true , address=45.91.116.12:4225 , piece_id=45AQV5XOE65GJNRXVLAUC4U6RVFEX674RVK46UNPEGVMO3K44BKQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.358503262 (628h39m48.229586297s) [seg_pos={0 0}, upload=550880]: Segment reached optimal threshold of 13 pieces. Dec 31 20:08:48.358683947 (628h39m48.229766982s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 31 20:08:48.358690960 (628h39m48.229773995s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358804706 (628h39m48.229887741s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection Dec 31 20:08:48.358809816 (628h39m48.229892851s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358916048 (628h39m48.229999083s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 20:08:48.358921628 (628h39m48.230004674s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358941036 (628h39m48.230024071s) [seg_pos={0 0}, upload=550880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:08:48.359006771 (628h39m48.230089806s) [upload=550880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:08:48.359011350 (628h39m48.230094395s) [upload=550880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:08:48.656600718 (628h39m48.527683753s) [upload=550880]: Done uploading segments... Dec 31 20:08:48.656620145 (628h39m48.52770319s) [upload=550880]: Upload finished. err: <nil> Duration: 7.224128391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.......XXXXXX.......XX..X.X.........XX.X..X......XX.....X.] duration: 6.09092013s trace: https://jaeger.dev.storj.io/trace/22b3ade1c92d50df FAIL (2025-12-31 20:09:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 31 20:08:45.626736104 (628h39m45.497819149s) [upload=550880]: Starting upload Dec 31 20:08:45.626804665 (628h39m45.49788771s) [upload=550880]: Done starting upload Dec 31 20:08:45.627061393 (628h39m45.498144438s) [upload=550880]: Uploading segments... Dec 31 20:08:45.627250494 (628h39m45.498333559s) [upload=550880]: Got next segment. Inline: false Dec 31 20:08:45.627323052 (628h39m45.498406107s) [upload=550880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:08:45.627351256 (628h39m45.498434321s) [upload=550880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:08:45.873963948 (628h39m45.745046983s) [seg_pos={0 0}, upload=550880]: Begin upload segment... Dec 31 20:08:45.874006550 (628h39m45.745089585s) [seg_pos={0 0}, upload=550880]: Done begin upload segment. Dec 31 20:08:45.874030585 (628h39m45.74511362s) [noise=true , address=145.79.227.4:4186 , piece_id=YH52AJGA3ETRW4FUO7SPFT7H2XKTDF5BO36DARZZHIBJ3PRFSC6Q, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874059239 (628h39m45.745142274s) [noise=true , address=145.79.227.15:4874 , piece_id=ELPCBAXMTDB32TABI36AN7K4RUIP6BVJW7EX6NSI62TN77RHSSMA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874417953 (628h39m45.745500988s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874685572 (628h39m45.745768607s) [noise=true , address=145.79.227.45:4231 , piece_id=63NGKSNJKGAILN6ECPJ7D45C7I7HAREF4WEGFQJAI42RNWZ5EU7Q, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874994280 (628h39m45.746077315s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875380014 (628h39m45.746463059s) [noise=true , address=145.79.227.20:4438 , piece_id=T6OG3TGALBDRD67QYTQJA2WOM7TFAXMQ6PU6QNF27YJV6UYX7Z2A, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875708851 (628h39m45.746791886s) [noise=true , address=45.91.116.8:4046 , piece_id=63TKYBTHKGRTGZT3BS5VEKOCRB4EV3BFTY22GUE7TWRYGJNS2ZXQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875976731 (628h39m45.747059766s) [noise=true , address=145.79.227.29:4941 , piece_id=QN6D7CPS2J6CGMOPLVNE7YCDYB5ALUJD2WRJYVKCWQKX7AZD72HQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.876333430 (628h39m45.747416466s) [noise=true , address=145.79.227.34:4906 , piece_id=XY3TADM333BZ5HJH3XR6SG63AS54JYRIM4CMBFZRI6YQOQBUJLZA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.877685394 (628h39m45.748768429s) [noise=true , address=77.90.31.21:4880 , piece_id=KLFJG2YW2MQXII6RDBKDPVXFX4BNQMMJ7SSMBG5OFTUFPCFY4TDA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.877986307 (628h39m45.749069342s) [noise=true , address=145.79.227.48:4302 , piece_id=43BZYTZ6SCK22662YBAAPNETKJVMBXISI6VRKZHZJQA5VODD57SA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.879698287 (628h39m45.750781332s) [noise=true , address=145.79.227.3:4151 , piece_id=V2FU7IRTRBMPGMWFSNCZLNSQW5UIKCSDOVBFPCQ7PG2HK6UUHVTA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.880797090 (628h39m45.751880125s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.881937301 (628h39m45.753020336s) [noise=true , address=145.79.227.25:4619 , piece_id=PJLHQRFUQS3XK2432Z7HVCB2FM5XARRF3EUANJWUBL56KOOX36DA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.883129030 (628h39m45.754212075s) [noise=true , address=45.91.116.12:4225 , piece_id=45AQV5XOE65GJNRXVLAUC4U6RVFEX674RVK46UNPEGVMO3K44BKQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.884236879 (628h39m45.755319914s) [noise=true , address=145.79.227.17:4972 , piece_id=ILGIZC2DKXFCOFRR5RVRVUNQNL5ASA2WKARMIJINQFF4LLRTVTJA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:47.580126643 (628h39m47.451209688s) [upload=550880]: Next returned nil segment Dec 31 20:08:47.580142973 (628h39m47.451226018s) [upload=550880]: Waiting for error group managing segments... Dec 31 20:08:47.866248295 (628h39m47.73733134s) [noise=true , address=145.79.227.25:4619 , piece_id=PJLHQRFUQS3XK2432Z7HVCB2FM5XARRF3EUANJWUBL56KOOX36DA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.869725967 (628h39m47.740809013s) [noise=true , address=145.79.227.15:4874 , piece_id=ELPCBAXMTDB32TABI36AN7K4RUIP6BVJW7EX6NSI62TN77RHSSMA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.870460967 (628h39m47.741544002s) [noise=true , address=145.79.227.3:4151 , piece_id=V2FU7IRTRBMPGMWFSNCZLNSQW5UIKCSDOVBFPCQ7PG2HK6UUHVTA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.871616026 (628h39m47.742699071s) [noise=true , address=145.79.227.45:4231 , piece_id=63NGKSNJKGAILN6ECPJ7D45C7I7HAREF4WEGFQJAI42RNWZ5EU7Q, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.871892693 (628h39m47.742975758s) [noise=true , address=145.79.227.20:4438 , piece_id=T6OG3TGALBDRD67QYTQJA2WOM7TFAXMQ6PU6QNF27YJV6UYX7Z2A, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.887991537 (628h39m47.759074572s) [noise=true , address=145.79.227.4:4186 , piece_id=YH52AJGA3ETRW4FUO7SPFT7H2XKTDF5BO36DARZZHIBJ3PRFSC6Q, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.898635435 (628h39m47.76971848s) [noise=true , address=145.79.227.17:4972 , piece_id=ILGIZC2DKXFCOFRR5RVRVUNQNL5ASA2WKARMIJINQFF4LLRTVTJA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.922822599 (628h39m47.793905634s) [noise=true , address=145.79.227.29:4941 , piece_id=QN6D7CPS2J6CGMOPLVNE7YCDYB5ALUJD2WRJYVKCWQKX7AZD72HQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.950286805 (628h39m47.82136984s) [noise=true , address=145.79.227.48:4302 , piece_id=43BZYTZ6SCK22662YBAAPNETKJVMBXISI6VRKZHZJQA5VODD57SA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.058506780 (628h39m47.929589845s) [noise=true , address=77.90.31.21:4880 , piece_id=KLFJG2YW2MQXII6RDBKDPVXFX4BNQMMJ7SSMBG5OFTUFPCFY4TDA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.176175496 (628h39m48.047258531s) [noise=true , address=45.91.116.8:4046 , piece_id=63TKYBTHKGRTGZT3BS5VEKOCRB4EV3BFTY22GUE7TWRYGJNS2ZXQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.295916799 (628h39m48.166999844s) [noise=true , address=145.79.227.34:4906 , piece_id=XY3TADM333BZ5HJH3XR6SG63AS54JYRIM4CMBFZRI6YQOQBUJLZA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.358484216 (628h39m48.229567261s) [noise=true , address=45.91.116.12:4225 , piece_id=45AQV5XOE65GJNRXVLAUC4U6RVFEX674RVK46UNPEGVMO3K44BKQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.358503262 (628h39m48.229586297s) [seg_pos={0 0}, upload=550880]: Segment reached optimal threshold of 13 pieces. Dec 31 20:08:48.358683947 (628h39m48.229766982s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 31 20:08:48.358690960 (628h39m48.229773995s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358804706 (628h39m48.229887741s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection Dec 31 20:08:48.358809816 (628h39m48.229892851s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358916048 (628h39m48.229999083s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 20:08:48.358921628 (628h39m48.230004674s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358941036 (628h39m48.230024071s) [seg_pos={0 0}, upload=550880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:08:48.359006771 (628h39m48.230089806s) [upload=550880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:08:48.359011350 (628h39m48.230094395s) [upload=550880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:08:48.656600718 (628h39m48.527683753s) [upload=550880]: Done uploading segments... Dec 31 20:08:48.656620145 (628h39m48.52770319s) [upload=550880]: Upload finished. err: <nil> Duration: 6.090953444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 29.371728846s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:09:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 29.371775335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X.......XXXXXX.......XX..X.X.........XX.X..X......XX.....X] duration: 30.002709608s trace: https://jaeger.dev.storj.io/trace/22b3ade1c92d50df FAIL (2025-12-31 20:09:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 31 20:08:45.626736104 (628h39m45.497819149s) [upload=550880]: Starting upload Dec 31 20:08:45.626804665 (628h39m45.49788771s) [upload=550880]: Done starting upload Dec 31 20:08:45.627061393 (628h39m45.498144438s) [upload=550880]: Uploading segments... Dec 31 20:08:45.627250494 (628h39m45.498333559s) [upload=550880]: Got next segment. Inline: false Dec 31 20:08:45.627323052 (628h39m45.498406107s) [upload=550880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:08:45.627351256 (628h39m45.498434321s) [upload=550880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:08:45.873963948 (628h39m45.745046983s) [seg_pos={0 0}, upload=550880]: Begin upload segment... Dec 31 20:08:45.874006550 (628h39m45.745089585s) [seg_pos={0 0}, upload=550880]: Done begin upload segment. Dec 31 20:08:45.874030585 (628h39m45.74511362s) [noise=true , address=145.79.227.4:4186 , piece_id=YH52AJGA3ETRW4FUO7SPFT7H2XKTDF5BO36DARZZHIBJ3PRFSC6Q, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874059239 (628h39m45.745142274s) [noise=true , address=145.79.227.15:4874 , piece_id=ELPCBAXMTDB32TABI36AN7K4RUIP6BVJW7EX6NSI62TN77RHSSMA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874417953 (628h39m45.745500988s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874685572 (628h39m45.745768607s) [noise=true , address=145.79.227.45:4231 , piece_id=63NGKSNJKGAILN6ECPJ7D45C7I7HAREF4WEGFQJAI42RNWZ5EU7Q, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.874994280 (628h39m45.746077315s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875380014 (628h39m45.746463059s) [noise=true , address=145.79.227.20:4438 , piece_id=T6OG3TGALBDRD67QYTQJA2WOM7TFAXMQ6PU6QNF27YJV6UYX7Z2A, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875708851 (628h39m45.746791886s) [noise=true , address=45.91.116.8:4046 , piece_id=63TKYBTHKGRTGZT3BS5VEKOCRB4EV3BFTY22GUE7TWRYGJNS2ZXQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.875976731 (628h39m45.747059766s) [noise=true , address=145.79.227.29:4941 , piece_id=QN6D7CPS2J6CGMOPLVNE7YCDYB5ALUJD2WRJYVKCWQKX7AZD72HQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.876333430 (628h39m45.747416466s) [noise=true , address=145.79.227.34:4906 , piece_id=XY3TADM333BZ5HJH3XR6SG63AS54JYRIM4CMBFZRI6YQOQBUJLZA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.877685394 (628h39m45.748768429s) [noise=true , address=77.90.31.21:4880 , piece_id=KLFJG2YW2MQXII6RDBKDPVXFX4BNQMMJ7SSMBG5OFTUFPCFY4TDA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.877986307 (628h39m45.749069342s) [noise=true , address=145.79.227.48:4302 , piece_id=43BZYTZ6SCK22662YBAAPNETKJVMBXISI6VRKZHZJQA5VODD57SA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.879698287 (628h39m45.750781332s) [noise=true , address=145.79.227.3:4151 , piece_id=V2FU7IRTRBMPGMWFSNCZLNSQW5UIKCSDOVBFPCQ7PG2HK6UUHVTA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.880797090 (628h39m45.751880125s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.881937301 (628h39m45.753020336s) [noise=true , address=145.79.227.25:4619 , piece_id=PJLHQRFUQS3XK2432Z7HVCB2FM5XARRF3EUANJWUBL56KOOX36DA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.883129030 (628h39m45.754212075s) [noise=true , address=45.91.116.12:4225 , piece_id=45AQV5XOE65GJNRXVLAUC4U6RVFEX674RVK46UNPEGVMO3K44BKQ, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:45.884236879 (628h39m45.755319914s) [noise=true , address=145.79.227.17:4972 , piece_id=ILGIZC2DKXFCOFRR5RVRVUNQNL5ASA2WKARMIJINQFF4LLRTVTJA, seg_pos={0 0}, upload=550880]: Uploading piece... Dec 31 20:08:47.580126643 (628h39m47.451209688s) [upload=550880]: Next returned nil segment Dec 31 20:08:47.580142973 (628h39m47.451226018s) [upload=550880]: Waiting for error group managing segments... Dec 31 20:08:47.866248295 (628h39m47.73733134s) [noise=true , address=145.79.227.25:4619 , piece_id=PJLHQRFUQS3XK2432Z7HVCB2FM5XARRF3EUANJWUBL56KOOX36DA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.869725967 (628h39m47.740809013s) [noise=true , address=145.79.227.15:4874 , piece_id=ELPCBAXMTDB32TABI36AN7K4RUIP6BVJW7EX6NSI62TN77RHSSMA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.870460967 (628h39m47.741544002s) [noise=true , address=145.79.227.3:4151 , piece_id=V2FU7IRTRBMPGMWFSNCZLNSQW5UIKCSDOVBFPCQ7PG2HK6UUHVTA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.871616026 (628h39m47.742699071s) [noise=true , address=145.79.227.45:4231 , piece_id=63NGKSNJKGAILN6ECPJ7D45C7I7HAREF4WEGFQJAI42RNWZ5EU7Q, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.871892693 (628h39m47.742975758s) [noise=true , address=145.79.227.20:4438 , piece_id=T6OG3TGALBDRD67QYTQJA2WOM7TFAXMQ6PU6QNF27YJV6UYX7Z2A, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.887991537 (628h39m47.759074572s) [noise=true , address=145.79.227.4:4186 , piece_id=YH52AJGA3ETRW4FUO7SPFT7H2XKTDF5BO36DARZZHIBJ3PRFSC6Q, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.898635435 (628h39m47.76971848s) [noise=true , address=145.79.227.17:4972 , piece_id=ILGIZC2DKXFCOFRR5RVRVUNQNL5ASA2WKARMIJINQFF4LLRTVTJA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.922822599 (628h39m47.793905634s) [noise=true , address=145.79.227.29:4941 , piece_id=QN6D7CPS2J6CGMOPLVNE7YCDYB5ALUJD2WRJYVKCWQKX7AZD72HQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:47.950286805 (628h39m47.82136984s) [noise=true , address=145.79.227.48:4302 , piece_id=43BZYTZ6SCK22662YBAAPNETKJVMBXISI6VRKZHZJQA5VODD57SA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.058506780 (628h39m47.929589845s) [noise=true , address=77.90.31.21:4880 , piece_id=KLFJG2YW2MQXII6RDBKDPVXFX4BNQMMJ7SSMBG5OFTUFPCFY4TDA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.176175496 (628h39m48.047258531s) [noise=true , address=45.91.116.8:4046 , piece_id=63TKYBTHKGRTGZT3BS5VEKOCRB4EV3BFTY22GUE7TWRYGJNS2ZXQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.295916799 (628h39m48.166999844s) [noise=true , address=145.79.227.34:4906 , piece_id=XY3TADM333BZ5HJH3XR6SG63AS54JYRIM4CMBFZRI6YQOQBUJLZA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.358484216 (628h39m48.229567261s) [noise=true , address=45.91.116.12:4225 , piece_id=45AQV5XOE65GJNRXVLAUC4U6RVFEX674RVK46UNPEGVMO3K44BKQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: <nil> Dec 31 20:08:48.358503262 (628h39m48.229586297s) [seg_pos={0 0}, upload=550880]: Segment reached optimal threshold of 13 pieces. Dec 31 20:08:48.358683947 (628h39m48.229766982s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 31 20:08:48.358690960 (628h39m48.229773995s) [noise=true , address=145.79.227.35:4482 , piece_id=UWKTBO4XAMMXKL7K2IVX7FXIQAXNM7RGOVY7A2W5GF24ZKECHTUA, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358804706 (628h39m48.229887741s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->77.90.31.13:4627: use of closed network connection Dec 31 20:08:48.358809816 (628h39m48.229892851s) [noise=true , address=77.90.31.13:4627 , piece_id=ABUI3RLKQEYEYYHXL5LMORQBWNRZBSLIVERNMOPJX5RKGG7BB4YQ, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358916048 (628h39m48.229999083s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 20:08:48.358921628 (628h39m48.230004674s) [noise=true , address=45.91.116.6:4975 , piece_id=QA44DD5TRV22GJUIP2ADAEA55FE65BHA6R3T3PI22QWWF2TPPRQA, seg_pos={0 0}, upload=550880]: Long tail cancellation due to optimal threshold Dec 31 20:08:48.358941036 (628h39m48.230024071s) [seg_pos={0 0}, upload=550880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:08:48.359006771 (628h39m48.230089806s) [upload=550880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:08:48.359011350 (628h39m48.230094395s) [upload=550880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:08:48.656600718 (628h39m48.527683753s) [upload=550880]: Done uploading segments... Dec 31 20:08:48.656620145 (628h39m48.52770319s) [upload=550880]: Upload finished. err: <nil> Duration: 30.00272545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 25.031255597s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:08:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 25.031362021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 15.441138037s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:07:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.441162423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X.......XXXXXX.......XX..X.X.........XX.X..X......XX.....] duration: 6.075046095s trace: https://jaeger.dev.storj.io/trace/4fd1f32594d71532 FAIL (2025-12-31 20:07:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A) context deadline exceeded ======LOGS====== Dec 31 20:03:57.913075397 (628h34m57.784158442s) [upload=550814]: Starting upload Dec 31 20:03:57.913175086 (628h34m57.784258182s) [upload=550814]: Done starting upload Dec 31 20:03:57.913323008 (628h34m57.784406053s) [upload=550814]: Uploading segments... Dec 31 20:03:57.913524352 (628h34m57.784607387s) [upload=550814]: Got next segment. Inline: false Dec 31 20:03:57.913566702 (628h34m57.784649747s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:03:57.913572363 (628h34m57.784655408s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:03:58.159552240 (628h34m58.030635285s) [seg_pos={0 0}, upload=550814]: Begin upload segment... Dec 31 20:03:58.159598497 (628h34m58.030681532s) [seg_pos={0 0}, upload=550814]: Done begin upload segment. Dec 31 20:03:58.159624507 (628h34m58.030707542s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.159714459 (628h34m58.030797504s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160433417 (628h34m58.031516452s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160684486 (628h34m58.031767521s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160761302 (628h34m58.031844347s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160819363 (628h34m58.031902398s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160859930 (628h34m58.031942965s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160911698 (628h34m58.031994733s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160944110 (628h34m58.032027145s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.161001299 (628h34m58.032084344s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162022584 (628h34m58.033105659s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162380765 (628h34m58.03346381s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162615172 (628h34m58.033698217s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163078624 (628h34m58.034161659s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163565892 (628h34m58.034648927s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163968549 (628h34m58.035051584s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:59.865244889 (628h34m59.736327934s) [upload=550814]: Next returned nil segment Dec 31 20:03:59.865259868 (628h34m59.736342913s) [upload=550814]: Waiting for error group managing segments... Dec 31 20:04:00.148434058 (628h35m0.019517103s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.154237809 (628h35m0.025320855s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.161003252 (628h35m0.032086287s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.180931217 (628h35m0.052014252s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.182368663 (628h35m0.053451708s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186053300 (628h35m0.057136335s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186338633 (628h35m0.057421678s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.199359687 (628h35m0.070442722s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.467593158 (628h35m0.338676203s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.567015100 (628h35m0.438098145s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.580262224 (628h35m0.451345269s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.806026900 (628h35m0.677109935s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.909993354 (628h35m0.781076389s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.910035294 (628h35m0.781118339s) [seg_pos={0 0}, upload=550814]: Segment reached optimal threshold of 13 pieces. Dec 31 20:04:00.910312412 (628h35m0.781395457s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910323894 (628h35m0.781406939s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910364481 (628h35m0.781447526s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910374690 (628h35m0.781457735s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910540576 (628h35m0.781623612s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910548581 (628h35m0.781631627s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910568420 (628h35m0.781651465s) [seg_pos={0 0}, upload=550814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:04:00.910647891 (628h35m0.781730946s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:04:00.910655926 (628h35m0.781738991s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:04:01.209277991 (628h35m1.080361036s) [upload=550814]: Done uploading segments... Dec 31 20:04:01.209298000 (628h35m1.080381035s) [upload=550814]: Upload finished. err: <nil> Duration: 6.075072094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 15.509834506s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:07:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.509859423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X.......XXXXXX.......XX..X.X.........XX.X..X......XX....] duration: 10.946379669s trace: https://jaeger.dev.storj.io/trace/4fd1f32594d71532 FAIL (2025-12-31 20:07:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A) context deadline exceeded ======LOGS====== Dec 31 20:03:57.913075397 (628h34m57.784158442s) [upload=550814]: Starting upload Dec 31 20:03:57.913175086 (628h34m57.784258182s) [upload=550814]: Done starting upload Dec 31 20:03:57.913323008 (628h34m57.784406053s) [upload=550814]: Uploading segments... Dec 31 20:03:57.913524352 (628h34m57.784607387s) [upload=550814]: Got next segment. Inline: false Dec 31 20:03:57.913566702 (628h34m57.784649747s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:03:57.913572363 (628h34m57.784655408s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:03:58.159552240 (628h34m58.030635285s) [seg_pos={0 0}, upload=550814]: Begin upload segment... Dec 31 20:03:58.159598497 (628h34m58.030681532s) [seg_pos={0 0}, upload=550814]: Done begin upload segment. Dec 31 20:03:58.159624507 (628h34m58.030707542s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.159714459 (628h34m58.030797504s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160433417 (628h34m58.031516452s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160684486 (628h34m58.031767521s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160761302 (628h34m58.031844347s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160819363 (628h34m58.031902398s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160859930 (628h34m58.031942965s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160911698 (628h34m58.031994733s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160944110 (628h34m58.032027145s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.161001299 (628h34m58.032084344s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162022584 (628h34m58.033105659s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162380765 (628h34m58.03346381s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162615172 (628h34m58.033698217s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163078624 (628h34m58.034161659s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163565892 (628h34m58.034648927s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163968549 (628h34m58.035051584s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:59.865244889 (628h34m59.736327934s) [upload=550814]: Next returned nil segment Dec 31 20:03:59.865259868 (628h34m59.736342913s) [upload=550814]: Waiting for error group managing segments... Dec 31 20:04:00.148434058 (628h35m0.019517103s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.154237809 (628h35m0.025320855s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.161003252 (628h35m0.032086287s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.180931217 (628h35m0.052014252s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.182368663 (628h35m0.053451708s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186053300 (628h35m0.057136335s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186338633 (628h35m0.057421678s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.199359687 (628h35m0.070442722s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.467593158 (628h35m0.338676203s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.567015100 (628h35m0.438098145s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.580262224 (628h35m0.451345269s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.806026900 (628h35m0.677109935s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.909993354 (628h35m0.781076389s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.910035294 (628h35m0.781118339s) [seg_pos={0 0}, upload=550814]: Segment reached optimal threshold of 13 pieces. Dec 31 20:04:00.910312412 (628h35m0.781395457s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910323894 (628h35m0.781406939s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910364481 (628h35m0.781447526s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910374690 (628h35m0.781457735s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910540576 (628h35m0.781623612s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910548581 (628h35m0.781631627s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910568420 (628h35m0.781651465s) [seg_pos={0 0}, upload=550814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:04:00.910647891 (628h35m0.781730946s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:04:00.910655926 (628h35m0.781738991s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:04:01.209277991 (628h35m1.080361036s) [upload=550814]: Done uploading segments... Dec 31 20:04:01.209298000 (628h35m1.080381035s) [upload=550814]: Upload finished. err: <nil> Duration: 10.946404927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 22.672057762s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:06:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 22.672088322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX.X.......XXXXXX.......XX..X.X.........XX.X..X......XX...] duration: 6.015580259s trace: https://jaeger.dev.storj.io/trace/4fd1f32594d71532 FAIL (2025-12-31 20:06:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A) context deadline exceeded ======LOGS====== Dec 31 20:03:57.913075397 (628h34m57.784158442s) [upload=550814]: Starting upload Dec 31 20:03:57.913175086 (628h34m57.784258182s) [upload=550814]: Done starting upload Dec 31 20:03:57.913323008 (628h34m57.784406053s) [upload=550814]: Uploading segments... Dec 31 20:03:57.913524352 (628h34m57.784607387s) [upload=550814]: Got next segment. Inline: false Dec 31 20:03:57.913566702 (628h34m57.784649747s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:03:57.913572363 (628h34m57.784655408s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:03:58.159552240 (628h34m58.030635285s) [seg_pos={0 0}, upload=550814]: Begin upload segment... Dec 31 20:03:58.159598497 (628h34m58.030681532s) [seg_pos={0 0}, upload=550814]: Done begin upload segment. Dec 31 20:03:58.159624507 (628h34m58.030707542s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.159714459 (628h34m58.030797504s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160433417 (628h34m58.031516452s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160684486 (628h34m58.031767521s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160761302 (628h34m58.031844347s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160819363 (628h34m58.031902398s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160859930 (628h34m58.031942965s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160911698 (628h34m58.031994733s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160944110 (628h34m58.032027145s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.161001299 (628h34m58.032084344s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162022584 (628h34m58.033105659s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162380765 (628h34m58.03346381s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162615172 (628h34m58.033698217s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163078624 (628h34m58.034161659s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163565892 (628h34m58.034648927s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163968549 (628h34m58.035051584s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:59.865244889 (628h34m59.736327934s) [upload=550814]: Next returned nil segment Dec 31 20:03:59.865259868 (628h34m59.736342913s) [upload=550814]: Waiting for error group managing segments... Dec 31 20:04:00.148434058 (628h35m0.019517103s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.154237809 (628h35m0.025320855s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.161003252 (628h35m0.032086287s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.180931217 (628h35m0.052014252s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.182368663 (628h35m0.053451708s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186053300 (628h35m0.057136335s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186338633 (628h35m0.057421678s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.199359687 (628h35m0.070442722s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.467593158 (628h35m0.338676203s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.567015100 (628h35m0.438098145s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.580262224 (628h35m0.451345269s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.806026900 (628h35m0.677109935s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.909993354 (628h35m0.781076389s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.910035294 (628h35m0.781118339s) [seg_pos={0 0}, upload=550814]: Segment reached optimal threshold of 13 pieces. Dec 31 20:04:00.910312412 (628h35m0.781395457s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910323894 (628h35m0.781406939s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910364481 (628h35m0.781447526s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910374690 (628h35m0.781457735s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910540576 (628h35m0.781623612s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910548581 (628h35m0.781631627s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910568420 (628h35m0.781651465s) [seg_pos={0 0}, upload=550814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:04:00.910647891 (628h35m0.781730946s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:04:00.910655926 (628h35m0.781738991s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:04:01.209277991 (628h35m1.080361036s) [upload=550814]: Done uploading segments... Dec 31 20:04:01.209298000 (628h35m1.080381035s) [upload=550814]: Upload finished. err: <nil> Duration: 6.015612221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX.X.......XXXXXX.......XX..X.X.........XX.X..X......XX..] duration: 6.059397097s trace: https://jaeger.dev.storj.io/trace/4fd1f32594d71532 FAIL (2025-12-31 20:05:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A) context deadline exceeded ======LOGS====== Dec 31 20:03:57.913075397 (628h34m57.784158442s) [upload=550814]: Starting upload Dec 31 20:03:57.913175086 (628h34m57.784258182s) [upload=550814]: Done starting upload Dec 31 20:03:57.913323008 (628h34m57.784406053s) [upload=550814]: Uploading segments... Dec 31 20:03:57.913524352 (628h34m57.784607387s) [upload=550814]: Got next segment. Inline: false Dec 31 20:03:57.913566702 (628h34m57.784649747s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:03:57.913572363 (628h34m57.784655408s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:03:58.159552240 (628h34m58.030635285s) [seg_pos={0 0}, upload=550814]: Begin upload segment... Dec 31 20:03:58.159598497 (628h34m58.030681532s) [seg_pos={0 0}, upload=550814]: Done begin upload segment. Dec 31 20:03:58.159624507 (628h34m58.030707542s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.159714459 (628h34m58.030797504s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160433417 (628h34m58.031516452s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160684486 (628h34m58.031767521s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160761302 (628h34m58.031844347s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160819363 (628h34m58.031902398s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160859930 (628h34m58.031942965s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160911698 (628h34m58.031994733s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160944110 (628h34m58.032027145s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.161001299 (628h34m58.032084344s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162022584 (628h34m58.033105659s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162380765 (628h34m58.03346381s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162615172 (628h34m58.033698217s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163078624 (628h34m58.034161659s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163565892 (628h34m58.034648927s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163968549 (628h34m58.035051584s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:59.865244889 (628h34m59.736327934s) [upload=550814]: Next returned nil segment Dec 31 20:03:59.865259868 (628h34m59.736342913s) [upload=550814]: Waiting for error group managing segments... Dec 31 20:04:00.148434058 (628h35m0.019517103s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.154237809 (628h35m0.025320855s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.161003252 (628h35m0.032086287s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.180931217 (628h35m0.052014252s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.182368663 (628h35m0.053451708s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186053300 (628h35m0.057136335s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186338633 (628h35m0.057421678s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.199359687 (628h35m0.070442722s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.467593158 (628h35m0.338676203s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.567015100 (628h35m0.438098145s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.580262224 (628h35m0.451345269s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.806026900 (628h35m0.677109935s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.909993354 (628h35m0.781076389s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.910035294 (628h35m0.781118339s) [seg_pos={0 0}, upload=550814]: Segment reached optimal threshold of 13 pieces. Dec 31 20:04:00.910312412 (628h35m0.781395457s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910323894 (628h35m0.781406939s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910364481 (628h35m0.781447526s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910374690 (628h35m0.781457735s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910540576 (628h35m0.781623612s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910548581 (628h35m0.781631627s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910568420 (628h35m0.781651465s) [seg_pos={0 0}, upload=550814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:04:00.910647891 (628h35m0.781730946s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:04:00.910655926 (628h35m0.781738991s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:04:01.209277991 (628h35m1.080361036s) [upload=550814]: Done uploading segments... Dec 31 20:04:01.209298000 (628h35m1.080381035s) [upload=550814]: Upload finished. err: <nil> Duration: 6.059427684s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.456555957s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:05:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.456585373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...XX.X.......XXXXXX.......XX..X.X.........XX.X..X......XX.] duration: 21.144589449s trace: https://jaeger.dev.storj.io/trace/4fd1f32594d71532 FAIL (2025-12-31 20:04:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A) context deadline exceeded ======LOGS====== Dec 31 20:03:57.913075397 (628h34m57.784158442s) [upload=550814]: Starting upload Dec 31 20:03:57.913175086 (628h34m57.784258182s) [upload=550814]: Done starting upload Dec 31 20:03:57.913323008 (628h34m57.784406053s) [upload=550814]: Uploading segments... Dec 31 20:03:57.913524352 (628h34m57.784607387s) [upload=550814]: Got next segment. Inline: false Dec 31 20:03:57.913566702 (628h34m57.784649747s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:03:57.913572363 (628h34m57.784655408s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:03:58.159552240 (628h34m58.030635285s) [seg_pos={0 0}, upload=550814]: Begin upload segment... Dec 31 20:03:58.159598497 (628h34m58.030681532s) [seg_pos={0 0}, upload=550814]: Done begin upload segment. Dec 31 20:03:58.159624507 (628h34m58.030707542s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.159714459 (628h34m58.030797504s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160433417 (628h34m58.031516452s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160684486 (628h34m58.031767521s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160761302 (628h34m58.031844347s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160819363 (628h34m58.031902398s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160859930 (628h34m58.031942965s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160911698 (628h34m58.031994733s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160944110 (628h34m58.032027145s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.161001299 (628h34m58.032084344s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162022584 (628h34m58.033105659s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162380765 (628h34m58.03346381s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162615172 (628h34m58.033698217s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163078624 (628h34m58.034161659s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163565892 (628h34m58.034648927s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163968549 (628h34m58.035051584s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:59.865244889 (628h34m59.736327934s) [upload=550814]: Next returned nil segment Dec 31 20:03:59.865259868 (628h34m59.736342913s) [upload=550814]: Waiting for error group managing segments... Dec 31 20:04:00.148434058 (628h35m0.019517103s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.154237809 (628h35m0.025320855s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.161003252 (628h35m0.032086287s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.180931217 (628h35m0.052014252s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.182368663 (628h35m0.053451708s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186053300 (628h35m0.057136335s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186338633 (628h35m0.057421678s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.199359687 (628h35m0.070442722s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.467593158 (628h35m0.338676203s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.567015100 (628h35m0.438098145s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.580262224 (628h35m0.451345269s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.806026900 (628h35m0.677109935s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.909993354 (628h35m0.781076389s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.910035294 (628h35m0.781118339s) [seg_pos={0 0}, upload=550814]: Segment reached optimal threshold of 13 pieces. Dec 31 20:04:00.910312412 (628h35m0.781395457s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910323894 (628h35m0.781406939s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910364481 (628h35m0.781447526s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910374690 (628h35m0.781457735s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910540576 (628h35m0.781623612s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910548581 (628h35m0.781631627s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910568420 (628h35m0.781651465s) [seg_pos={0 0}, upload=550814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:04:00.910647891 (628h35m0.781730946s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:04:00.910655926 (628h35m0.781738991s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:04:01.209277991 (628h35m1.080361036s) [upload=550814]: Done uploading segments... Dec 31 20:04:01.209298000 (628h35m1.080381035s) [upload=550814]: Upload finished. err: <nil> Duration: 21.144605652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X......XX] duration: 30.002433634s trace: https://jaeger.dev.storj.io/trace/4fd1f32594d71532 FAIL (2025-12-31 20:04:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A) context deadline exceeded ======LOGS====== Dec 31 20:03:57.913075397 (628h34m57.784158442s) [upload=550814]: Starting upload Dec 31 20:03:57.913175086 (628h34m57.784258182s) [upload=550814]: Done starting upload Dec 31 20:03:57.913323008 (628h34m57.784406053s) [upload=550814]: Uploading segments... Dec 31 20:03:57.913524352 (628h34m57.784607387s) [upload=550814]: Got next segment. Inline: false Dec 31 20:03:57.913566702 (628h34m57.784649747s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:03:57.913572363 (628h34m57.784655408s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:03:58.159552240 (628h34m58.030635285s) [seg_pos={0 0}, upload=550814]: Begin upload segment... Dec 31 20:03:58.159598497 (628h34m58.030681532s) [seg_pos={0 0}, upload=550814]: Done begin upload segment. Dec 31 20:03:58.159624507 (628h34m58.030707542s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.159714459 (628h34m58.030797504s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160433417 (628h34m58.031516452s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160684486 (628h34m58.031767521s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160761302 (628h34m58.031844347s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160819363 (628h34m58.031902398s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160859930 (628h34m58.031942965s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160911698 (628h34m58.031994733s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.160944110 (628h34m58.032027145s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.161001299 (628h34m58.032084344s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162022584 (628h34m58.033105659s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162380765 (628h34m58.03346381s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.162615172 (628h34m58.033698217s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163078624 (628h34m58.034161659s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163565892 (628h34m58.034648927s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:58.163968549 (628h34m58.035051584s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Uploading piece... Dec 31 20:03:59.865244889 (628h34m59.736327934s) [upload=550814]: Next returned nil segment Dec 31 20:03:59.865259868 (628h34m59.736342913s) [upload=550814]: Waiting for error group managing segments... Dec 31 20:04:00.148434058 (628h35m0.019517103s) [noise=true , address=145.79.227.19:4077 , piece_id=D4SFSRAP4223X3AVKS5CF2SSYB2U6KFNTDSINPA3ZQ2FEQJTN7FQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.154237809 (628h35m0.025320855s) [noise=true , address=145.79.227.41:4087 , piece_id=MNHGD2IEZQKRQQM6XP3HXFRMRVHOP7GXA2IM76A4CGEIL3YWGQHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.161003252 (628h35m0.032086287s) [noise=true , address=145.79.227.42:4121 , piece_id=W3DB3PWKH3VUQ677FZ5MF4FKLAXHYQWKGYURFEEJPVOHDT2EYCHA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.180931217 (628h35m0.052014252s) [noise=true , address=145.79.227.39:4013 , piece_id=I64LUFLXWC7H75VZMB5AJIP6U3IL5MEU6BNENS4IZG7M6FQJRGIQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.182368663 (628h35m0.053451708s) [noise=true , address=145.79.227.48:4303 , piece_id=MTWRZGFTKAJ37Y5S4E5A4RWQJFBDY6FINFS32GNAQXHVTXKW223Q, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186053300 (628h35m0.057136335s) [noise=true , address=145.79.227.50:4373 , piece_id=S32PBYJUIVHKVISTXS2KTXW7MZL75TMPIEIA2TJUKBBXJCSTILLA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.186338633 (628h35m0.057421678s) [noise=true , address=145.79.227.20:4437 , piece_id=BJCFT5NFMBE2G4CLOYHNQQTQLMHTEUZ6OEZA2YKBLMNNCGGLHLCA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.199359687 (628h35m0.070442722s) [noise=true , address=145.79.227.37:4565 , piece_id=J2YV7KATG4JTFZ4ZUNKLQ4YNTQNGSPJUXMLDCEQHINZVG7QCODXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.467593158 (628h35m0.338676203s) [noise=true , address=45.91.116.14:4192 , piece_id=7WWNCYE2P6XENCK2ZD66T3YWGZWRQ2IWEPCJFRTRBSTSNX4XFT5A, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.567015100 (628h35m0.438098145s) [noise=true , address=45.91.116.4:4939 , piece_id=TGOK22ROV6Z7Y34QQMBB7V6CCSX2FGNY7XANPFE6UR6IBVC2UEOQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.580262224 (628h35m0.451345269s) [noise=true , address=45.91.116.10:4083 , piece_id=IU2TN5ZBTABVWLCRYPBODXRHVLEVR7SRHJP2YEIR4YYUXCVQZLQA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.806026900 (628h35m0.677109935s) [noise=true , address=45.91.116.8:4047 , piece_id=3FOHZZC6C4K2IGN6OTR556XGJYI6CJH567UV52AJYVWKNA7JUNNA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.909993354 (628h35m0.781076389s) [noise=true , address=45.91.116.13:4154 , piece_id=BS5ZYX77ICFSVBSNWN77NOK4HOZNSOXFWBPD5U7ULPIN6RHPHQXA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: <nil> Dec 31 20:04:00.910035294 (628h35m0.781118339s) [seg_pos={0 0}, upload=550814]: Segment reached optimal threshold of 13 pieces. Dec 31 20:04:00.910312412 (628h35m0.781395457s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910323894 (628h35m0.781406939s) [noise=true , address=45.91.116.7:4010 , piece_id=MCYHO2PAIAL6S44A32TO46NGAH2QAHTW7JUHD5SUGLDRAQCVNEYA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910364481 (628h35m0.781447526s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910374690 (628h35m0.781457735s) [noise=true , address=45.91.116.11:4119 , piece_id=YM5QKJ5FJRTSTJYW52FLA55ZCBEJ2SAFUIWW7K7NL5FKFBTEN6OA, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910540576 (628h35m0.781623612s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 31 20:04:00.910548581 (628h35m0.781631627s) [noise=true , address=77.90.31.17:4773 , piece_id=LCSKJZRQFTA7CMELQEMSR6GO4CDO7EIOFEAPBIOCEZJDTQZWJ2RQ, seg_pos={0 0}, upload=550814]: Long tail cancellation due to optimal threshold Dec 31 20:04:00.910568420 (628h35m0.781651465s) [seg_pos={0 0}, upload=550814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:04:00.910647891 (628h35m0.781730946s) [upload=550814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:04:00.910655926 (628h35m0.781738991s) [upload=550814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:04:01.209277991 (628h35m1.080361036s) [upload=550814]: Done uploading segments... Dec 31 20:04:01.209298000 (628h35m1.080381035s) [upload=550814]: Upload finished. err: <nil> Duration: 30.002453151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 14.719240703s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:04:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 14.71926608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X......X] duration: 30.002333305s trace: https://jaeger.dev.storj.io/trace/6c1fb130b435e787 FAIL (2025-12-31 20:03:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: YNXKEY4OXT26PGOH6HCEALJ6RRTPTOXZA3XSUFYDPIHX733L4FXA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: A6FWAADAOQ3ZSWBVFU3KVFGGUKMWHPEA3LMDKSKGOKZX3MCN3YZQ) context deadline exceeded ======LOGS====== Dec 31 20:03:20.033888255 (628h34m19.90497129s) [upload=550805]: Starting upload Dec 31 20:03:20.033955534 (628h34m19.905038569s) [upload=550805]: Done starting upload Dec 31 20:03:20.034160734 (628h34m19.905243779s) [upload=550805]: Uploading segments... Dec 31 20:03:20.034198697 (628h34m19.905281732s) [upload=550805]: Got next segment. Inline: false Dec 31 20:03:20.034223965 (628h34m19.905307s) [upload=550805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:03:20.034228413 (628h34m19.905311448s) [upload=550805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:03:20.279117264 (628h34m20.150200309s) [seg_pos={0 0}, upload=550805]: Begin upload segment... Dec 31 20:03:20.279248604 (628h34m20.15033165s) [seg_pos={0 0}, upload=550805]: Done begin upload segment. Dec 31 20:03:20.279325972 (628h34m20.150409007s) [noise=true , address=45.91.116.4:4937 , piece_id=NUC436TLUVSQRYVUI2RVN3I7NM7MRI4SVMZETSH5IUSYGC36HXBQ, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.280280790 (628h34m20.151363825s) [noise=true , address=145.79.227.50:4373 , piece_id=Y2LBTL4SHE3W4RFNBN2IF5AU4EWPUZQWECLKE23QOQE6FLEBA36Q, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.280692814 (628h34m20.151775849s) [noise=true , address=45.91.116.14:4190 , piece_id=YNXKEY4OXT26PGOH6HCEALJ6RRTPTOXZA3XSUFYDPIHX733L4FXA, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.280969892 (628h34m20.152052947s) [noise=true , address=77.90.31.14:4664 , piece_id=A6FWAADAOQ3ZSWBVFU3KVFGGUKMWHPEA3LMDKSKGOKZX3MCN3YZQ, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.281272077 (628h34m20.152355112s) [noise=true , address=145.79.227.48:4301 , piece_id=OAVMPWTTNWU5243MFXHVKUQHWEEMT26ZHIIVTNZ7U7QSI5LMSM5A, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.281614339 (628h34m20.152697384s) [noise=true , address=77.90.31.11:4554 , piece_id=YZWFL7IEOS5TXYCMZVX4NMYSVQN5ZXE6FQAYREBTO4TOARV6A6OA, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.282179004 (628h34m20.153262039s) [noise=true , address=45.91.116.9:4262 , piece_id=ZXGIAAHFOYBCY47QIGDBJEQNZ6P5US47F2A6J2GWRORDM43TXDVQ, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.282602491 (628h34m20.153685526s) [noise=true , address=145.79.227.52:4446 , piece_id=VD5NCWWFTT65V2GLGAEZGUMVEJ22OGPBZMHWJGJGCQGAB63NV4BA, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.283076413 (628h34m20.154159458s) [noise=true , address=45.91.116.6:4974 , piece_id=4HQE67HU7RHRQVUOMQ37K37ZBHXK7E4XJSGCLAWUDSDHKJAUYSAA, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.283401151 (628h34m20.154484196s) [noise=true , address=145.79.227.46:4603 , piece_id=YGVM2WFERQJGJMTX5IAYMPGSDQEYMZKCBJBGLU2MRIZE5TAMU4LA, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.283730349 (628h34m20.154813384s) [noise=true , address=45.91.116.11:4120 , piece_id=EJMFNRJOA4IDLQBJXXXGTLD7VKZDVGRSXQUMX3JFCOABDS6U6WTA, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.284259346 (628h34m20.155342381s) [noise=true , address=45.91.116.5:4300 , piece_id=OGRHQH76VSVVPMXLQKI2EAMB7LDACZFBVMB6LIHJUQUR6JVY2VDQ, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.284921236 (628h34m20.156004271s) [noise=true , address=77.90.31.15:4700 , piece_id=HIUFHCJFHWFJJV4PS2BODZAVCP6HMF47CGJYHENYS4HV6AXEU6BQ, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.285250232 (628h34m20.156333277s) [noise=true , address=145.79.227.34:4906 , piece_id=4Z6ZF4LPZN4YIPTKNVWKLEGSLEUMXGHUELTZ5YKGPB3ABWUKJCAA, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.286035819 (628h34m20.157118854s) [noise=true , address=145.79.227.47:4267 , piece_id=NGCPZHQY3SOU5EHQ5D3HPBYX2I7R4C6KHKMDHMCX2RNBHBC2LLOQ, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:20.286440939 (628h34m20.157523974s) [noise=true , address=145.79.227.39:4015 , piece_id=EUTCV73UJEVN5HQN5TGVWGJ5JRE4LLWCH3HBRQWGUIFAFDY6AWUQ, seg_pos={0 0}, upload=550805]: Uploading piece... Dec 31 20:03:21.991814079 (628h34m21.862897125s) [upload=550805]: Next returned nil segment Dec 31 20:03:21.991830291 (628h34m21.862913336s) [upload=550805]: Waiting for error group managing segments... Dec 31 20:03:22.273924164 (628h34m22.145007209s) [noise=true , address=145.79.227.46:4603 , piece_id=YGVM2WFERQJGJMTX5IAYMPGSDQEYMZKCBJBGLU2MRIZE5TAMU4LA, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.290383895 (628h34m22.16146697s) [noise=true , address=145.79.227.48:4301 , piece_id=OAVMPWTTNWU5243MFXHVKUQHWEEMT26ZHIIVTNZ7U7QSI5LMSM5A, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.293710609 (628h34m22.164793664s) [noise=true , address=145.79.227.52:4446 , piece_id=VD5NCWWFTT65V2GLGAEZGUMVEJ22OGPBZMHWJGJGCQGAB63NV4BA, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.294227013 (628h34m22.165310048s) [noise=true , address=145.79.227.50:4373 , piece_id=Y2LBTL4SHE3W4RFNBN2IF5AU4EWPUZQWECLKE23QOQE6FLEBA36Q, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.310092603 (628h34m22.181175638s) [noise=true , address=145.79.227.47:4267 , piece_id=NGCPZHQY3SOU5EHQ5D3HPBYX2I7R4C6KHKMDHMCX2RNBHBC2LLOQ, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.313323124 (628h34m22.184406169s) [noise=true , address=145.79.227.39:4015 , piece_id=EUTCV73UJEVN5HQN5TGVWGJ5JRE4LLWCH3HBRQWGUIFAFDY6AWUQ, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.364513156 (628h34m22.235596191s) [noise=true , address=145.79.227.34:4906 , piece_id=4Z6ZF4LPZN4YIPTKNVWKLEGSLEUMXGHUELTZ5YKGPB3ABWUKJCAA, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.428781572 (628h34m22.299864607s) [noise=true , address=77.90.31.15:4700 , piece_id=HIUFHCJFHWFJJV4PS2BODZAVCP6HMF47CGJYHENYS4HV6AXEU6BQ, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.460452518 (628h34m22.331535563s) [noise=true , address=77.90.31.14:4664 , piece_id=A6FWAADAOQ3ZSWBVFU3KVFGGUKMWHPEA3LMDKSKGOKZX3MCN3YZQ, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.565764455 (628h34m22.43684749s) [noise=true , address=45.91.116.4:4937 , piece_id=NUC436TLUVSQRYVUI2RVN3I7NM7MRI4SVMZETSH5IUSYGC36HXBQ, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.586657248 (628h34m22.457740283s) [noise=true , address=45.91.116.11:4120 , piece_id=EJMFNRJOA4IDLQBJXXXGTLD7VKZDVGRSXQUMX3JFCOABDS6U6WTA, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.613957300 (628h34m22.485040336s) [noise=true , address=45.91.116.14:4190 , piece_id=YNXKEY4OXT26PGOH6HCEALJ6RRTPTOXZA3XSUFYDPIHX733L4FXA, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.659769753 (628h34m22.530852788s) [noise=true , address=45.91.116.5:4300 , piece_id=OGRHQH76VSVVPMXLQKI2EAMB7LDACZFBVMB6LIHJUQUR6JVY2VDQ, seg_pos={0 0}, upload=550805]: Done uploading piece. err: <nil> Dec 31 20:03:22.659786084 (628h34m22.530869119s) [seg_pos={0 0}, upload=550805]: Segment reached optimal threshold of 13 pieces. Dec 31 20:03:22.659975766 (628h34m22.531058801s) [noise=true , address=45.91.116.6:4974 , piece_id=4HQE67HU7RHRQVUOMQ37K37ZBHXK7E4XJSGCLAWUDSDHKJAUYSAA, seg_pos={0 0}, upload=550805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 20:03:22.659981486 (628h34m22.531064531s) [noise=true , address=45.91.116.6:4974 , piece_id=4HQE67HU7RHRQVUOMQ37K37ZBHXK7E4XJSGCLAWUDSDHKJAUYSAA, seg_pos={0 0}, upload=550805]: Long tail cancellation due to optimal threshold Dec 31 20:03:22.660188080 (628h34m22.531271115s) [noise=true , address=77.90.31.11:4554 , piece_id=YZWFL7IEOS5TXYCMZVX4NMYSVQN5ZXE6FQAYREBTO4TOARV6A6OA, seg_pos={0 0}, upload=550805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 31 20:03:22.660195674 (628h34m22.531278709s) [noise=true , address=77.90.31.11:4554 , piece_id=YZWFL7IEOS5TXYCMZVX4NMYSVQN5ZXE6FQAYREBTO4TOARV6A6OA, seg_pos={0 0}, upload=550805]: Long tail cancellation due to optimal threshold Dec 31 20:03:22.660375366 (628h34m22.531458411s) [noise=true , address=45.91.116.9:4262 , piece_id=ZXGIAAHFOYBCY47QIGDBJEQNZ6P5US47F2A6J2GWRORDM43TXDVQ, seg_pos={0 0}, upload=550805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 31 20:03:22.660381959 (628h34m22.531464994s) [noise=true , address=45.91.116.9:4262 , piece_id=ZXGIAAHFOYBCY47QIGDBJEQNZ6P5US47F2A6J2GWRORDM43TXDVQ, seg_pos={0 0}, upload=550805]: Long tail cancellation due to optimal threshold Dec 31 20:03:22.660395184 (628h34m22.531478219s) [seg_pos={0 0}, upload=550805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:03:22.660455339 (628h34m22.531538374s) [upload=550805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:03:22.660460298 (628h34m22.531543333s) [upload=550805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:03:22.951270698 (628h34m22.822353744s) [upload=550805]: Done uploading segments... Dec 31 20:03:22.951304032 (628h34m22.822387067s) [upload=550805]: Upload finished. err: <nil> Duration: 30.002356549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 8.221538062s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 8.22156342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........X..................................................] duration: 21.764454685s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 21.764491515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X......] duration: 24.954686767s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 20:02:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 24.954712996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........X.................................................] duration: 6.634401316s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:02:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 6.634427206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........X................................................] duration: 14.970201006s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 14.970228068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X.....] duration: 8.58095114s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 20:01:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 8.580978943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........X...............................................] duration: 15.303101589s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:01:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.303118912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X........X..............................................] duration: 27.524157284s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 20:00:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 27.524185688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X....] duration: 5.914385826s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 20:00:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 5.914425302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X...] duration: 28.061541641s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 20:00:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 28.061573733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X........X.............................................] duration: 18.228991766s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:59:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 18.229017948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X..] duration: 6.107345536s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 19:59:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 6.107380092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X........X............................................] duration: 8.072895432s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:58:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 8.072918505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X........X...........................................] duration: 22.25824774s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:58:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 22.258279451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X.] duration: 7.899838474s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 19:57:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 7.899875134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X........X..........................................] duration: 15.367744391s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:57:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.367768908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..X] duration: 30.002875049s trace: https://jaeger.dev.storj.io/trace/17d15f80836b53a7 FAIL (2025-12-31 19:57:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A) context deadline exceeded ======LOGS====== Dec 31 19:56:39.179915977 (628h27m39.050999012s) [upload=550708]: Starting upload Dec 31 19:56:39.179969198 (628h27m39.051052233s) [upload=550708]: Done starting upload Dec 31 19:56:39.180166484 (628h27m39.051249519s) [upload=550708]: Uploading segments... Dec 31 19:56:39.180312862 (628h27m39.051395907s) [upload=550708]: Got next segment. Inline: false Dec 31 19:56:39.180354651 (628h27m39.051437697s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:56:39.180361685 (628h27m39.05144473s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:56:39.424796942 (628h27m39.295879977s) [seg_pos={0 0}, upload=550708]: Begin upload segment... Dec 31 19:56:39.424847889 (628h27m39.295930924s) [seg_pos={0 0}, upload=550708]: Done begin upload segment. Dec 31 19:56:39.424879870 (628h27m39.295962915s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.424934634 (628h27m39.296017669s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425580444 (628h27m39.296663489s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.425677859 (628h27m39.296760904s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426162612 (628h27m39.297245657s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426584776 (628h27m39.297667811s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.426644370 (628h27m39.297727405s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427035584 (628h27m39.298118629s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427042939 (628h27m39.298125974s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427426129 (628h27m39.298509164s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427456316 (628h27m39.298539361s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427776656 (628h27m39.298859691s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.427904098 (628h27m39.298987133s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428138765 (628h27m39.2992218s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428333897 (628h27m39.299416942s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:39.428566980 (628h27m39.299650025s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Uploading piece... Dec 31 19:56:41.131309573 (628h27m41.002392608s) [upload=550708]: Next returned nil segment Dec 31 19:56:41.131324471 (628h27m41.002407516s) [upload=550708]: Waiting for error group managing segments... Dec 31 19:56:41.411231971 (628h27m41.282315016s) [noise=true , address=145.79.227.7:4293 , piece_id=DHM6HN774M3SGO5XVGX3SRFYUNSCQLKZW7SNPFF2A6MREODDWDUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.416212635 (628h27m41.28729567s) [noise=true , address=145.79.227.18:4006 , piece_id=NFBTBCSJDY2XTVQ2PIMVOPSKTWJRWUMH52URIYX2CH4YCBXU2YIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.423814109 (628h27m41.294897144s) [noise=true , address=145.79.227.26:4654 , piece_id=DDFZGBBEZBHGUC57KLMZKKPFV6TJJFL7E4SJDX5XZHL75YEKOJRQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.426529139 (628h27m41.297612174s) [noise=true , address=145.79.227.36:4518 , piece_id=UNLPN4TNNBPINR7Q6S7G5TIKK6CB2HDV2ERIGPWAQJ27CEPP3HUA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.430921693 (628h27m41.302004728s) [noise=true , address=145.79.227.9:4367 , piece_id=URXYQBIIECDP7KTGPWPWYPJELTGUZFNVTRFBILQRM336PT3NF7BA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434181199 (628h27m41.305264244s) [noise=true , address=145.79.227.46:4602 , piece_id=2AW6SMQZM3DQ6GWTVOZA3QUQO3D26USKTCLSVYZ3QKIBLTCQCVNA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.434678456 (628h27m41.305761501s) [noise=true , address=145.79.227.13:4804 , piece_id=R4UQ4RF66HKFLUWUNVY6HHEZ4TWQHIY6RECIRK3TXQCBWLSWCXWQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.446974329 (628h27m41.318057364s) [noise=true , address=145.79.227.2:4115 , piece_id=GN7WHUL35MXXP7CDXMB7IXAAO6O7ROJGHN7ZIPGI7LKEFUZORAVQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.476437761 (628h27m41.347520796s) [noise=true , address=145.79.227.12:4767 , piece_id=GIKOXUOHFTIDTLHFOQHKMRH6E46SBV52LOPXNAC5GYSFAJK47N2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.584335051 (628h27m41.455418096s) [noise=true , address=77.90.31.19:4841 , piece_id=TX76PFC5JYCG6M7NB524KTUP3BL4TZZ6FLVMNKY647BXXYVDK2EA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.602116890 (628h27m41.473199935s) [noise=true , address=77.90.31.21:4878 , piece_id=VOM6XDWUEOXB77Z447OAW6WNZRPC6WBPVTCBELPAO2OZFUIXEAMQ, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:41.908974066 (628h27m41.780057111s) [noise=true , address=77.90.31.12:4589 , piece_id=XJSCD6EYFEIGCGIAFW5GXTUZXAUAXGQFIUGUV74R4RY3GRDWQS4Q, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103260603 (628h27m41.974343638s) [noise=true , address=45.91.116.6:4975 , piece_id=SOIPJY3ICVKYTKUCNWD3M2GUREIQ6XIWLUFXLL5NBVWTLX3M3R2A, seg_pos={0 0}, upload=550708]: Done uploading piece. err: <nil> Dec 31 19:56:42.103280099 (628h27m41.974363134s) [seg_pos={0 0}, upload=550708]: Segment reached optimal threshold of 13 pieces. Dec 31 19:56:42.103452177 (628h27m41.974535212s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103459190 (628h27m41.974542225s) [noise=true , address=145.79.227.44:4194 , piece_id=PSSODTMRGREMW3HNQSCFAP6URWKVU3KEHSHIYCOGJNCVXDFFFPSA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103535035 (628h27m41.97461807s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103542379 (628h27m41.974625414s) [noise=true , address=145.79.227.49:4339 , piece_id=WOECKPOZPXQCHFVVV4DRYB4GTOHKYPHE343Y3JU4UB6MYB57RVIA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103651717 (628h27m41.974734752s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:56:42.103657428 (628h27m41.974740463s) [noise=true , address=77.90.31.14:4664 , piece_id=DQI7I6G6OTUHRWBSMSRIGZFC4JPO34ROIAPLYA52QURKUPNF73QA, seg_pos={0 0}, upload=550708]: Long tail cancellation due to optimal threshold Dec 31 19:56:42.103676565 (628h27m41.9747596s) [seg_pos={0 0}, upload=550708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:56:42.103758460 (628h27m41.974841495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:56:42.103766465 (628h27m41.9748495s) [upload=550708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:56:42.404432642 (628h27m42.275515678s) [upload=550708]: Done uploading segments... Dec 31 19:56:42.404450587 (628h27m42.275533622s) [upload=550708]: Upload finished. err: <nil> Duration: 30.002898574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X........X.........................................] duration: 15.935156518s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:56:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 15.935170604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X........X........................................] duration: 56.451118318s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:56:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 56.451149737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X..] duration: 8.881804071s trace: https://jaeger.dev.storj.io/trace/2f715d2301976f5b FAIL (2025-12-31 19:55:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:54:08.872929961 (628h25m8.744013006s) [upload=550674]: Starting upload Dec 31 19:54:08.872980828 (628h25m8.744063863s) [upload=550674]: Done starting upload Dec 31 19:54:08.873182172 (628h25m8.744265207s) [upload=550674]: Uploading segments... Dec 31 19:54:08.873220575 (628h25m8.74430362s) [upload=550674]: Got next segment. Inline: false Dec 31 19:54:08.873248347 (628h25m8.744331382s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:54:08.873254199 (628h25m8.744337244s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:54:09.120276862 (628h25m8.991359897s) [seg_pos={0 0}, upload=550674]: Begin upload segment... Dec 31 19:54:09.120322499 (628h25m8.991405534s) [seg_pos={0 0}, upload=550674]: Done begin upload segment. Dec 31 19:54:09.120346555 (628h25m8.99142959s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.122488714 (628h25m8.993571749s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123139543 (628h25m8.994222588s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123705310 (628h25m8.994788355s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124142132 (628h25m8.995225178s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124561220 (628h25m8.995644255s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125114454 (628h25m8.996197489s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125540806 (628h25m8.996623841s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125989530 (628h25m8.997072565s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.127497261 (628h25m8.998580296s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128016329 (628h25m8.999099364s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128764093 (628h25m8.999847128s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129354998 (628h25m9.000438033s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129752676 (628h25m9.000835711s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130331117 (628h25m9.001414152s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130795962 (628h25m9.001879007s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:10.828192543 (628h25m10.699275578s) [upload=550674]: Next returned nil segment Dec 31 19:54:10.828205367 (628h25m10.699288412s) [upload=550674]: Waiting for error group managing segments... Dec 31 19:54:11.113750290 (628h25m10.984833325s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.125464996 (628h25m10.996548031s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.133658137 (628h25m11.004741172s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.135268614 (628h25m11.00635166s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.142300114 (628h25m11.013383149s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.145157574 (628h25m11.016240609s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.146337340 (628h25m11.017420375s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.196719965 (628h25m11.067803s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.443330843 (628h25m11.314413888s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.554534298 (628h25m11.425617333s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.600673403 (628h25m11.471756448s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.779758755 (628h25m11.6508418s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821242274 (628h25m11.692325309s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821263064 (628h25m11.692346099s) [seg_pos={0 0}, upload=550674]: Segment reached optimal threshold of 13 pieces. Dec 31 19:54:11.821428088 (628h25m11.692511124s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821434070 (628h25m11.692517115s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821538579 (628h25m11.692621624s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection Dec 31 19:54:11.821556923 (628h25m11.692639968s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821611077 (628h25m11.692694122s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821616868 (628h25m11.692699913s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821678976 (628h25m11.692762021s) [seg_pos={0 0}, upload=550674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:54:11.821785238 (628h25m11.692868283s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:54:11.821792782 (628h25m11.692875828s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:54:12.109997061 (628h25m11.981080106s) [upload=550674]: Done uploading segments... Dec 31 19:54:12.110010177 (628h25m11.981093212s) [upload=550674]: Upload finished. err: <nil> Duration: 8.881836221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X.] duration: 5.992913566s trace: https://jaeger.dev.storj.io/trace/2f715d2301976f5b FAIL (2025-12-31 19:55:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:54:08.872929961 (628h25m8.744013006s) [upload=550674]: Starting upload Dec 31 19:54:08.872980828 (628h25m8.744063863s) [upload=550674]: Done starting upload Dec 31 19:54:08.873182172 (628h25m8.744265207s) [upload=550674]: Uploading segments... Dec 31 19:54:08.873220575 (628h25m8.74430362s) [upload=550674]: Got next segment. Inline: false Dec 31 19:54:08.873248347 (628h25m8.744331382s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:54:08.873254199 (628h25m8.744337244s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:54:09.120276862 (628h25m8.991359897s) [seg_pos={0 0}, upload=550674]: Begin upload segment... Dec 31 19:54:09.120322499 (628h25m8.991405534s) [seg_pos={0 0}, upload=550674]: Done begin upload segment. Dec 31 19:54:09.120346555 (628h25m8.99142959s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.122488714 (628h25m8.993571749s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123139543 (628h25m8.994222588s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123705310 (628h25m8.994788355s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124142132 (628h25m8.995225178s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124561220 (628h25m8.995644255s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125114454 (628h25m8.996197489s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125540806 (628h25m8.996623841s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125989530 (628h25m8.997072565s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.127497261 (628h25m8.998580296s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128016329 (628h25m8.999099364s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128764093 (628h25m8.999847128s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129354998 (628h25m9.000438033s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129752676 (628h25m9.000835711s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130331117 (628h25m9.001414152s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130795962 (628h25m9.001879007s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:10.828192543 (628h25m10.699275578s) [upload=550674]: Next returned nil segment Dec 31 19:54:10.828205367 (628h25m10.699288412s) [upload=550674]: Waiting for error group managing segments... Dec 31 19:54:11.113750290 (628h25m10.984833325s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.125464996 (628h25m10.996548031s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.133658137 (628h25m11.004741172s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.135268614 (628h25m11.00635166s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.142300114 (628h25m11.013383149s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.145157574 (628h25m11.016240609s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.146337340 (628h25m11.017420375s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.196719965 (628h25m11.067803s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.443330843 (628h25m11.314413888s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.554534298 (628h25m11.425617333s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.600673403 (628h25m11.471756448s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.779758755 (628h25m11.6508418s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821242274 (628h25m11.692325309s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821263064 (628h25m11.692346099s) [seg_pos={0 0}, upload=550674]: Segment reached optimal threshold of 13 pieces. Dec 31 19:54:11.821428088 (628h25m11.692511124s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821434070 (628h25m11.692517115s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821538579 (628h25m11.692621624s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection Dec 31 19:54:11.821556923 (628h25m11.692639968s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821611077 (628h25m11.692694122s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821616868 (628h25m11.692699913s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821678976 (628h25m11.692762021s) [seg_pos={0 0}, upload=550674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:54:11.821785238 (628h25m11.692868283s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:54:11.821792782 (628h25m11.692875828s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:54:12.109997061 (628h25m11.981080106s) [upload=550674]: Done uploading segments... Dec 31 19:54:12.110010177 (628h25m11.981093212s) [upload=550674]: Upload finished. err: <nil> Duration: 5.992944416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X........X.......................................] duration: 17.96762238s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:55:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 17.967640686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........X......................................] duration: 27.284073964s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:54:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 27.284100394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.X] duration: 30.002318857s trace: https://jaeger.dev.storj.io/trace/2f715d2301976f5b FAIL (2025-12-31 19:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 19:54:08.872929961 (628h25m8.744013006s) [upload=550674]: Starting upload Dec 31 19:54:08.872980828 (628h25m8.744063863s) [upload=550674]: Done starting upload Dec 31 19:54:08.873182172 (628h25m8.744265207s) [upload=550674]: Uploading segments... Dec 31 19:54:08.873220575 (628h25m8.74430362s) [upload=550674]: Got next segment. Inline: false Dec 31 19:54:08.873248347 (628h25m8.744331382s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:54:08.873254199 (628h25m8.744337244s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:54:09.120276862 (628h25m8.991359897s) [seg_pos={0 0}, upload=550674]: Begin upload segment... Dec 31 19:54:09.120322499 (628h25m8.991405534s) [seg_pos={0 0}, upload=550674]: Done begin upload segment. Dec 31 19:54:09.120346555 (628h25m8.99142959s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.122488714 (628h25m8.993571749s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123139543 (628h25m8.994222588s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.123705310 (628h25m8.994788355s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124142132 (628h25m8.995225178s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.124561220 (628h25m8.995644255s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125114454 (628h25m8.996197489s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125540806 (628h25m8.996623841s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.125989530 (628h25m8.997072565s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.127497261 (628h25m8.998580296s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128016329 (628h25m8.999099364s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.128764093 (628h25m8.999847128s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129354998 (628h25m9.000438033s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.129752676 (628h25m9.000835711s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130331117 (628h25m9.001414152s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:09.130795962 (628h25m9.001879007s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Uploading piece... Dec 31 19:54:10.828192543 (628h25m10.699275578s) [upload=550674]: Next returned nil segment Dec 31 19:54:10.828205367 (628h25m10.699288412s) [upload=550674]: Waiting for error group managing segments... Dec 31 19:54:11.113750290 (628h25m10.984833325s) [noise=true , address=145.79.227.21:4474 , piece_id=3JOCLMA743WXWJUC4OSP3RYEBRICGEZLPPJZ4F2V3IUZC26H5U7Q, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.125464996 (628h25m10.996548031s) [noise=true , address=145.79.227.15:4874 , piece_id=PO3322DZMTDX2EQ2BECIAEQEZVZPAYGXWHIGQF2HC52G53WQH2WA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.133658137 (628h25m11.004741172s) [noise=true , address=145.79.227.24:4583 , piece_id=OW3TVQXFS5DQB7EBXJYN3UFSIFQV723FZPS3RKEJNHIT73C5GLRQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.135268614 (628h25m11.00635166s) [noise=true , address=145.79.227.20:4437 , piece_id=ZX5IWJAE2PAKWO42A3OZU7YSVGR4XVCY77APQH76YKFBQQO4S4HQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.142300114 (628h25m11.013383149s) [noise=true , address=145.79.227.52:4447 , piece_id=USTYL3SOM4NLRKPDSQFDWEFT5JR35SUJADXMDOTNNYLUB3IC2TYA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.145157574 (628h25m11.016240609s) [noise=true , address=145.79.227.51:4409 , piece_id=FZ3TB465IJN7PASBTXRVWRXFM6ABY5UIQXLK3ELXUTZ6RABNDETQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.146337340 (628h25m11.017420375s) [noise=true , address=145.79.227.31:4798 , piece_id=V2RBPHQXZB6ZASJ5ERSNQGWYKGWTPX3Y7LLPLLTW7ZM4G7AUWAQQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.196719965 (628h25m11.067803s) [noise=true , address=145.79.227.11:4729 , piece_id=YLEMVKH6LHPRLBXWN2LU4JJ66WI5LPOFEXCK3WNL5RAYSMXJ66IA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.443330843 (628h25m11.314413888s) [noise=true , address=45.91.116.10:4083 , piece_id=GPWGFSFVYF3UJNYHAAM6KIOX6CJMCXHUBTAD5OVNWZXAJXUZECGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.554534298 (628h25m11.425617333s) [noise=true , address=45.91.116.6:4976 , piece_id=MH7GIZZWRLOTTJRCQPZRHT6PXWHTHP3UIKK2ZPZ3ZDUXEWZUQ4DQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.600673403 (628h25m11.471756448s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWQEYCIOBWW4NMLWK7XB2OMIVLGQ7T53EQOCU7UZUB5CCHPSOOQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.779758755 (628h25m11.6508418s) [noise=true , address=45.91.116.9:4264 , piece_id=OFXH5MX4GDOO6AKYJESFMG7I4GPGO3X6RCTJGYFWHJUOLP3D2FGQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821242274 (628h25m11.692325309s) [noise=true , address=45.91.116.8:4045 , piece_id=32ZBNKSWJDJBYNETLNWJOFP532RX4ZL2VMKGGFJZ5QJPCWFPNDMQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: <nil> Dec 31 19:54:11.821263064 (628h25m11.692346099s) [seg_pos={0 0}, upload=550674]: Segment reached optimal threshold of 13 pieces. Dec 31 19:54:11.821428088 (628h25m11.692511124s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821434070 (628h25m11.692517115s) [noise=true , address=77.90.31.14:4664 , piece_id=X7RGBOWADRKTQ66FSHRDVKPPQOJX4C5I5BEME22LBVCSB5SXKZFQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821538579 (628h25m11.692621624s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35802->145.79.227.50:4373: use of closed network connection Dec 31 19:54:11.821556923 (628h25m11.692639968s) [noise=true , address=145.79.227.50:4373 , piece_id=M4OL5N62FE7TPQFMMXLKDYHA443NURCHBUKLETTNSDAM75NIEXTA, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821611077 (628h25m11.692694122s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 19:54:11.821616868 (628h25m11.692699913s) [noise=true , address=145.79.227.34:4907 , piece_id=IW5V4Y3WKFACIK7YJH5BSV6JHQV7VYOHTZLGN7J2O2RRDRQ47WKQ, seg_pos={0 0}, upload=550674]: Long tail cancellation due to optimal threshold Dec 31 19:54:11.821678976 (628h25m11.692762021s) [seg_pos={0 0}, upload=550674]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:54:11.821785238 (628h25m11.692868283s) [upload=550674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:54:11.821792782 (628h25m11.692875828s) [upload=550674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:54:12.109997061 (628h25m11.981080106s) [upload=550674]: Done uploading segments... Dec 31 19:54:12.110010177 (628h25m11.981093212s) [upload=550674]: Upload finished. err: <nil> Duration: 30.002340306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........X.....................................] duration: 29.639008812s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:54:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 29.639030903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........X....................................] duration: 28.743259539s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:53:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 28.743288023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX.] duration: 5.989878541s trace: https://jaeger.dev.storj.io/trace/7a8a51965de90f4d FAIL (2025-12-31 19:53:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA) context deadline exceeded ======LOGS====== 1 19:52:18.817657431 (628h23m18.688740466s) [upload=550647]: Starting upload Dec 31 19:52:18.817714298 (628h23m18.688797333s) [upload=550647]: Done starting upload Dec 31 19:52:18.817877780 (628h23m18.688960835s) [upload=550647]: Uploading segments... Dec 31 19:52:18.818458766 (628h23m18.689541801s) [upload=550647]: Got next segment. Inline: false Dec 31 19:52:18.818512238 (628h23m18.689595273s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:52:18.818534160 (628h23m18.689617195s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:52:19.073533471 (628h23m18.944616616s) [seg_pos={0 0}, upload=550647]: Begin upload segment... Dec 31 19:52:19.073628232 (628h23m18.944711307s) [seg_pos={0 0}, upload=550647]: Done begin upload segment. Dec 31 19:52:19.073663018 (628h23m18.944746053s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075170148 (628h23m18.946253193s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075272603 (628h23m18.946355638s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075791371 (628h23m18.946874416s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076129034 (628h23m18.947212139s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076173939 (628h23m18.947256984s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076500360 (628h23m18.947583395s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076756589 (628h23m18.947839634s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076887237 (628h23m18.947970272s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077319000 (628h23m18.948402035s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077344107 (628h23m18.948427152s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077729911 (628h23m18.948812946s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077880609 (628h23m18.948963684s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.078487264 (628h23m18.949570309s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079006983 (628h23m18.950090029s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079466488 (628h23m18.950549533s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:20.787238688 (628h23m20.658321733s) [upload=550647]: Next returned nil segment Dec 31 19:52:20.787251121 (628h23m20.658334156s) [upload=550647]: Waiting for error group managing segments... Dec 31 19:52:21.067516362 (628h23m20.938599407s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.086302995 (628h23m20.95738603s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.092391418 (628h23m20.963474463s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.098140284 (628h23m20.969223329s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.101875838 (628h23m20.972958883s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.105102892 (628h23m20.976185937s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.151296661 (628h23m21.022379696s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.153107429 (628h23m21.024190464s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.165848730 (628h23m21.036931765s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.261047671 (628h23m21.132130716s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.270159400 (628h23m21.141242445s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.373973044 (628h23m21.245056089s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377309598 (628h23m21.248392683s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377335407 (628h23m21.248418452s) [seg_pos={0 0}, upload=550647]: Segment reached optimal threshold of 13 pieces. Dec 31 19:52:21.377610561 (628h23m21.248693606s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377620410 (628h23m21.248703455s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377917025 (628h23m21.24900006s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection Dec 31 19:52:21.377927515 (628h23m21.24901056s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377935209 (628h23m21.249018264s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377946581 (628h23m21.249029627s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.378058975 (628h23m21.24914202s) [seg_pos={0 0}, upload=550647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:52:21.378152432 (628h23m21.249235478s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:52:21.378159447 (628h23m21.249242492s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:52:21.667390809 (628h23m21.538473855s) [upload=550647]: Done uploading segments... Dec 31 19:52:21.667415386 (628h23m21.538498421s) [upload=550647]: Upload finished. err: <nil> Duration: 5.989904771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X........X...................................] duration: 25.240428965s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 25.240453713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........XX] duration: 30.001959663s trace: https://jaeger.dev.storj.io/trace/7a8a51965de90f4d FAIL (2025-12-31 19:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA) context deadline exceeded ======LOGS====== 1 19:52:18.817657431 (628h23m18.688740466s) [upload=550647]: Starting upload Dec 31 19:52:18.817714298 (628h23m18.688797333s) [upload=550647]: Done starting upload Dec 31 19:52:18.817877780 (628h23m18.688960835s) [upload=550647]: Uploading segments... Dec 31 19:52:18.818458766 (628h23m18.689541801s) [upload=550647]: Got next segment. Inline: false Dec 31 19:52:18.818512238 (628h23m18.689595273s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:52:18.818534160 (628h23m18.689617195s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:52:19.073533471 (628h23m18.944616616s) [seg_pos={0 0}, upload=550647]: Begin upload segment... Dec 31 19:52:19.073628232 (628h23m18.944711307s) [seg_pos={0 0}, upload=550647]: Done begin upload segment. Dec 31 19:52:19.073663018 (628h23m18.944746053s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075170148 (628h23m18.946253193s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075272603 (628h23m18.946355638s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.075791371 (628h23m18.946874416s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076129034 (628h23m18.947212139s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076173939 (628h23m18.947256984s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076500360 (628h23m18.947583395s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076756589 (628h23m18.947839634s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.076887237 (628h23m18.947970272s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077319000 (628h23m18.948402035s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077344107 (628h23m18.948427152s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077729911 (628h23m18.948812946s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.077880609 (628h23m18.948963684s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.078487264 (628h23m18.949570309s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079006983 (628h23m18.950090029s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:19.079466488 (628h23m18.950549533s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Uploading piece... Dec 31 19:52:20.787238688 (628h23m20.658321733s) [upload=550647]: Next returned nil segment Dec 31 19:52:20.787251121 (628h23m20.658334156s) [upload=550647]: Waiting for error group managing segments... Dec 31 19:52:21.067516362 (628h23m20.938599407s) [noise=true , address=145.79.227.7:4294 , piece_id=3O2O44PGJ2MTVM5VVB543O2IWSOCJPEBL246IYRDDFZ6ZVRO644Q, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.086302995 (628h23m20.95738603s) [noise=true , address=145.79.227.45:4231 , piece_id=Y23NALVPF36R45LZKV6W2UBOG4VEIWYLHUN5ZK75OLUUTMPQBAFA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.092391418 (628h23m20.963474463s) [noise=true , address=145.79.227.17:4971 , piece_id=ANZHU57WPYV4E4SRWV4EL6547SKZDJQZ7HU4PLL7BQT2JU65KJXA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.098140284 (628h23m20.969223329s) [noise=true , address=145.79.227.41:4086 , piece_id=HRGAGTAZ56ZYBQDOOGSZZPNDX77REHE4PHLSC4S4R4HXOVRVLXPQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.101875838 (628h23m20.972958883s) [noise=true , address=145.79.227.27:4691 , piece_id=FRMK4CEUGHDBQFGNFSQ3C3U5MZPTTKUEY4OTLNWZLFUS4D7Q5BGQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.105102892 (628h23m20.976185937s) [noise=true , address=145.79.227.21:4475 , piece_id=U4YEBCYSXM7QPFBECU4RN6EB435V3COFRSZOJNI5YNZ6OU4UZEEQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.151296661 (628h23m21.022379696s) [noise=true , address=145.79.227.51:4410 , piece_id=KPSYGU5DZF6TKQ74Q76GUGUBXM7KJVV7JJBCD2M6XQZJSE76F2KQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.153107429 (628h23m21.024190464s) [noise=true , address=145.79.227.22:4511 , piece_id=7MDQ4PGXVGIZLOS6A2D5M3WU3NHESTTLZDGVRQEF4T3DILYB2J5A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.165848730 (628h23m21.036931765s) [noise=true , address=145.79.227.3:4151 , piece_id=AIW6Z4V5XGSRF3OJOGRNZ2XE74DPERQFEVVHD753PYC3WUNZ2TVA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.261047671 (628h23m21.132130716s) [noise=true , address=77.90.31.11:4555 , piece_id=J6TMJHAZD44NVPTGQFUWJDXHK6JGJSQA3YVR3VMAFKRSSL2V6MQA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.270159400 (628h23m21.141242445s) [noise=true , address=77.90.31.14:4664 , piece_id=RK5E4F3VPW3VAXA5B2MIVH3G2K4QO4YHAK2D6UQ46ADEN74JJWAA, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.373973044 (628h23m21.245056089s) [noise=true , address=45.91.116.12:4226 , piece_id=3EIQBSK3JQF74XHMOJBRNKEJVRIKP5R7EILPH7AIO3DDVNQOKYKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377309598 (628h23m21.248392683s) [noise=true , address=45.91.116.13:4155 , piece_id=TQ723AKQ5L6IGZ2NMGQOP62UND2GKSTOWA4HV5XES7ZOVBPQTSVQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: <nil> Dec 31 19:52:21.377335407 (628h23m21.248418452s) [seg_pos={0 0}, upload=550647]: Segment reached optimal threshold of 13 pieces. Dec 31 19:52:21.377610561 (628h23m21.248693606s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377620410 (628h23m21.248703455s) [noise=true , address=45.91.116.4:4937 , piece_id=CQX3W63K33SPU6WNBSCOEPZYIUXQLWBP72K5OZAD2Z3IPOQYVX2A, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377917025 (628h23m21.24900006s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60370->45.91.116.9:4264: use of closed network connection Dec 31 19:52:21.377927515 (628h23m21.24901056s) [noise=true , address=45.91.116.9:4264 , piece_id=QNKKHH74QMYEFOBH74654ZF5MGWJOOM7CJ7YCJO4F3Q7SSFO5XKQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.377935209 (628h23m21.249018264s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 19:52:21.377946581 (628h23m21.249029627s) [noise=true , address=45.91.116.6:4974 , piece_id=NHDM4IM3P2LEVYYG7X5OTFJVLR46FJYS722JB3GCHDDJOYZ5G2AQ, seg_pos={0 0}, upload=550647]: Long tail cancellation due to optimal threshold Dec 31 19:52:21.378058975 (628h23m21.24914202s) [seg_pos={0 0}, upload=550647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:52:21.378152432 (628h23m21.249235478s) [upload=550647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:52:21.378159447 (628h23m21.249242492s) [upload=550647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:52:21.667390809 (628h23m21.538473855s) [upload=550647]: Done uploading segments... Dec 31 19:52:21.667415386 (628h23m21.538498421s) [upload=550647]: Upload finished. err: <nil> Duration: 30.00197418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........X] duration: 30.00381083s trace: https://jaeger.dev.storj.io/trace/6c41fe23c50dbad0 FAIL (2025-12-31 19:52:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: FBG4S26JMYO4SM3YRBACTUTL4BYVMTGN4YVMJ533LFEF7CZHK6YA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: REV4BOYOSHVF4CJKVZ3BAFDJWBSW64KRDRW5X3LHR5CQV423JEIQ) context deadline exceeded ======LOGS====== Dec 31 19:51:48.218191366 (628h22m48.089274401s) [upload=550639]: Starting upload Dec 31 19:51:48.218249015 (628h22m48.08933205s) [upload=550639]: Done starting upload Dec 31 19:51:48.218424068 (628h22m48.089507103s) [upload=550639]: Uploading segments... Dec 31 19:51:48.218473573 (628h22m48.089556608s) [upload=550639]: Got next segment. Inline: false Dec 31 19:51:48.218497859 (628h22m48.089580904s) [upload=550639]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:51:48.218502958 (628h22m48.089585993s) [upload=550639]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:51:48.463412829 (628h22m48.334495874s) [seg_pos={0 0}, upload=550639]: Begin upload segment... Dec 31 19:51:48.463474937 (628h22m48.334557972s) [seg_pos={0 0}, upload=550639]: Done begin upload segment. Dec 31 19:51:48.463528059 (628h22m48.334611104s) [noise=true , address=145.79.227.36:4517 , piece_id=TPETK4NVZAB7PELSEVNRIDWJMG47DULAQNWMJJD2RMFHA32WFVKQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.463533839 (628h22m48.334616874s) [noise=true , address=145.79.227.19:4079 , piece_id=VKBDOWGSA7DF5XUABZPU5D7QFJQZGLPY3BVMZCAG6C5DU7P47WTA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.463709794 (628h22m48.334792839s) [noise=true , address=145.79.227.4:4187 , piece_id=QKGVTQZS67EYQQQATPW6PHKK5Z22WGF3B4HG2GHFYBLLI2VXSJCA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464082584 (628h22m48.33516562s) [noise=true , address=45.91.116.13:4154 , piece_id=FBG4S26JMYO4SM3YRBACTUTL4BYVMTGN4YVMJ533LFEF7CZHK6YA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464448952 (628h22m48.335531997s) [noise=true , address=45.91.116.6:4975 , piece_id=SFRL4MYTZWAUIDSK2Q4QOIVQSZAYGINHU56LITP5IS7IRUW4ECMQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464585492 (628h22m48.335668537s) [noise=true , address=45.91.116.9:4264 , piece_id=REV4BOYOSHVF4CJKVZ3BAFDJWBSW64KRDRW5X3LHR5CQV423JEIQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.464893007 (628h22m48.335976042s) [noise=true , address=45.91.116.4:4940 , piece_id=ONKIN37ROFLVIUT4Q7WMY35OAT45K56AYOCSD5FLK3SLHT36ER7Q, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465154035 (628h22m48.33623707s) [noise=true , address=77.90.31.11:4556 , piece_id=HJ463DL6ZUSSGY77JBJCWNCSLDQNXC6F56K72QXGRI7UTU7CIW7Q, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465266469 (628h22m48.336349504s) [noise=true , address=145.79.227.34:4905 , piece_id=OZ253LCVPSZ64PKUCXFLGG6RYF7IMRYGA3IMYH3JLB2MK5RU43IA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465605435 (628h22m48.33668847s) [noise=true , address=145.79.227.17:4971 , piece_id=3KVX4NU43TJ2ZGIAAKLYMFZJOIGQHRGF3JCMCJVAS7PNTHRFABKA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.465626203 (628h22m48.336709249s) [noise=true , address=145.79.227.52:4447 , piece_id=7VSURSUZJOGBMRWJTVYQASRPWWFLDCYC6ZHNLNMGTWQOMX5E73MQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466033539 (628h22m48.337116584s) [noise=true , address=77.90.31.14:4663 , piece_id=FOH7IGII3MERETC6FWF2ZDMHE6VDQUYRQBF6KCSSFA4LFV3XT3PQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466039701 (628h22m48.337122736s) [noise=true , address=145.79.227.47:4266 , piece_id=Y4FNXLWJXFHX5R4BT7Q3SF5QP4OYTJM3PRVVBU2MBJ4OMGRKKI3Q, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466419464 (628h22m48.337502499s) [noise=true , address=145.79.227.10:4402 , piece_id=5JF5BEDDWHICPC76T3YVPOSTCJ46V73SZ4DLOLK7VRQVYHFIQEGA, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.466940877 (628h22m48.338023912s) [noise=true , address=145.79.227.45:4231 , piece_id=UTCXURTLGCJBE7GMVORZRTUA26ALJSZGEI4S6VXCJTSODOP2CPMQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:48.467401915 (628h22m48.33848496s) [noise=true , address=77.90.31.21:4878 , piece_id=57M6URN2OEZA6NTABQAHLPWU6F5ZGP2AZD4KKDAQ7ZUPNLMFE5XQ, seg_pos={0 0}, upload=550639]: Uploading piece... Dec 31 19:51:50.170694453 (628h22m50.041777498s) [upload=550639]: Next returned nil segment Dec 31 19:51:50.170720483 (628h22m50.041803538s) [upload=550639]: Waiting for error group managing segments... Dec 31 19:51:50.454371371 (628h22m50.325454406s) [noise=true , address=145.79.227.19:4079 , piece_id=VKBDOWGSA7DF5XUABZPU5D7QFJQZGLPY3BVMZCAG6C5DU7P47WTA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.456361590 (628h22m50.327444625s) [noise=true , address=145.79.227.4:4187 , piece_id=QKGVTQZS67EYQQQATPW6PHKK5Z22WGF3B4HG2GHFYBLLI2VXSJCA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.467506983 (628h22m50.338590028s) [noise=true , address=145.79.227.52:4447 , piece_id=7VSURSUZJOGBMRWJTVYQASRPWWFLDCYC6ZHNLNMGTWQOMX5E73MQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.468201415 (628h22m50.33928446s) [noise=true , address=145.79.227.10:4402 , piece_id=5JF5BEDDWHICPC76T3YVPOSTCJ46V73SZ4DLOLK7VRQVYHFIQEGA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.474112079 (628h22m50.345195114s) [noise=true , address=145.79.227.45:4231 , piece_id=UTCXURTLGCJBE7GMVORZRTUA26ALJSZGEI4S6VXCJTSODOP2CPMQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.475382468 (628h22m50.346465503s) [noise=true , address=145.79.227.47:4266 , piece_id=Y4FNXLWJXFHX5R4BT7Q3SF5QP4OYTJM3PRVVBU2MBJ4OMGRKKI3Q, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.478191186 (628h22m50.349274231s) [noise=true , address=145.79.227.36:4517 , piece_id=TPETK4NVZAB7PELSEVNRIDWJMG47DULAQNWMJJD2RMFHA32WFVKQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.495099462 (628h22m50.366182497s) [noise=true , address=145.79.227.17:4971 , piece_id=3KVX4NU43TJ2ZGIAAKLYMFZJOIGQHRGF3JCMCJVAS7PNTHRFABKA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.717526340 (628h22m50.588609386s) [noise=true , address=77.90.31.11:4556 , piece_id=HJ463DL6ZUSSGY77JBJCWNCSLDQNXC6F56K72QXGRI7UTU7CIW7Q, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.728020995 (628h22m50.59910404s) [noise=true , address=77.90.31.14:4663 , piece_id=FOH7IGII3MERETC6FWF2ZDMHE6VDQUYRQBF6KCSSFA4LFV3XT3PQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.729282246 (628h22m50.600365291s) [noise=true , address=77.90.31.21:4878 , piece_id=57M6URN2OEZA6NTABQAHLPWU6F5ZGP2AZD4KKDAQ7ZUPNLMFE5XQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.782546027 (628h22m50.653629062s) [noise=true , address=45.91.116.9:4264 , piece_id=REV4BOYOSHVF4CJKVZ3BAFDJWBSW64KRDRW5X3LHR5CQV423JEIQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.847626249 (628h22m50.718709284s) [noise=true , address=45.91.116.13:4154 , piece_id=FBG4S26JMYO4SM3YRBACTUTL4BYVMTGN4YVMJ533LFEF7CZHK6YA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: <nil> Dec 31 19:51:50.847645876 (628h22m50.718728921s) [seg_pos={0 0}, upload=550639]: Segment reached optimal threshold of 13 pieces. Dec 31 19:51:50.847818755 (628h22m50.71890179s) [noise=true , address=145.79.227.34:4905 , piece_id=OZ253LCVPSZ64PKUCXFLGG6RYF7IMRYGA3IMYH3JLB2MK5RU43IA, seg_pos={0 0}, upload=550639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 31 19:51:50.847830328 (628h22m50.718913363s) [noise=true , address=145.79.227.34:4905 , piece_id=OZ253LCVPSZ64PKUCXFLGG6RYF7IMRYGA3IMYH3JLB2MK5RU43IA, seg_pos={0 0}, upload=550639]: Long tail cancellation due to optimal threshold Dec 31 19:51:50.847940657 (628h22m50.719023703s) [noise=true , address=45.91.116.6:4975 , piece_id=SFRL4MYTZWAUIDSK2Q4QOIVQSZAYGINHU56LITP5IS7IRUW4ECMQ, seg_pos={0 0}, upload=550639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 19:51:50.847948693 (628h22m50.719031728s) [noise=true , address=45.91.116.6:4975 , piece_id=SFRL4MYTZWAUIDSK2Q4QOIVQSZAYGINHU56LITP5IS7IRUW4ECMQ, seg_pos={0 0}, upload=550639]: Long tail cancellation due to optimal threshold Dec 31 19:51:50.848089400 (628h22m50.719172435s) [noise=true , address=45.91.116.4:4940 , piece_id=ONKIN37ROFLVIUT4Q7WMY35OAT45K56AYOCSD5FLK3SLHT36ER7Q, seg_pos={0 0}, upload=550639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 19:51:50.848095191 (628h22m50.719178236s) [noise=true , address=45.91.116.4:4940 , piece_id=ONKIN37ROFLVIUT4Q7WMY35OAT45K56AYOCSD5FLK3SLHT36ER7Q, seg_pos={0 0}, upload=550639]: Long tail cancellation due to optimal threshold Dec 31 19:51:50.848110932 (628h22m50.719193967s) [seg_pos={0 0}, upload=550639]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:51:50.848189501 (628h22m50.719272546s) [upload=550639]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:51:50.848194971 (628h22m50.719278006s) [upload=550639]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:51:51.137679055 (628h22m51.00876209s) [upload=550639]: Done uploading segments... Dec 31 19:51:51.137700256 (628h22m51.008783291s) [upload=550639]: Upload finished. err: <nil> Duration: 30.003837841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........X..................................] duration: 27.452848552s trace: https://jaeger.dev.storj.io/trace/6f2db19365d5c4a7 FAIL (2025-12-31 19:52:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :25:11.407129282 (627h56m11.278212317s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.407653700 (627h56m11.278736746s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408090462 (627h56m11.279173507s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408532594 (627h56m11.279615639s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.408982150 (627h56m11.280065195s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409261151 (627h56m11.280344196s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409387272 (627h56m11.280470317s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409764960 (627h56m11.280848005s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.409855813 (627h56m11.280938848s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410206421 (627h56m11.281289466s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410434185 (627h56m11.28151723s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410576286 (627h56m11.281659321s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.410863622 (627h56m11.281946668s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411018307 (627h56m11.282101362s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411173443 (627h56m11.282256488s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.411517808 (627h56m11.282600853s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:11.414052164 (627h56m11.28513521s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Uploading piece... Dec 31 19:25:12.843531325 (627h56m12.71461442s) [upload=550237]: Next returned nil segment Dec 31 19:25:12.843553196 (627h56m12.714636231s) [upload=550237]: Waiting for error group managing segments... Dec 31 19:25:13.201602832 (627h56m13.072685867s) [noise=true , address=64.49.13.137:4501 , piece_id=UFKOH5EN4B5LWUYSRUYQMADKX4S6JXPM5ROLBFUKOOTSFZA3MCAA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.246118947 (627h56m13.117201982s) [noise=true , address=64.49.13.139:4478 , piece_id=OZAQFKGHM6MYHKBON42SSPC3XZOLZPGRGRP7VOYXGDF2OXN74BSA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.286549893 (627h56m13.157632928s) [noise=true , address=145.79.227.50:4388 , piece_id=E5S52J3R25L6GGSP3FC4EBMLXLY24UWUWH3DERWPEHM2TJ63KIJQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.295561943 (627h56m13.166644988s) [noise=true , address=145.79.227.51:4418 , piece_id=C5JPRZVHSGXPKQKBMZQV5MHLAS7OOAVXPKZVGX666MM7X3ZMQMIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.312131524 (627h56m13.183214569s) [noise=true , address=145.79.227.14:4855 , piece_id=PGLI2LID5C6SZWTTUQDD7GV6IWRHV6KRD3TTNF4QCHI3FE6S2CTA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.387843681 (627h56m13.258926716s) [noise=true , address=145.79.227.11:4744 , piece_id=4CUFWGQHNENEILRUJ6FC55Q5KBRZP4TUKDKPBRADXD5EMBXVZXGQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.464368143 (627h56m13.335451188s) [noise=true , address=77.90.31.19:4855 , piece_id=DCJ7O2ZPFKQVH22BHBGT56PU3CTOSAL7HXQG6QOZRQRARP2ZFM5Q, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.477254311 (627h56m13.348337356s) [noise=true , address=77.90.31.17:4781 , piece_id=2GASMVKCI43OABZ4BTK5JHHEDR2RJDUMGTF2N3SIYESPYPSNWUMA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.579867270 (627h56m13.450950305s) [noise=true , address=45.91.116.5:4323 , piece_id=K5SSPPQFF243G6GZOQTV5XYC6WABJ73OHTL5MC5FHYWVK6GT3JPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:13.674923139 (627h56m13.546006174s) [noise=true , address=45.91.116.7:4044 , piece_id=ECYWIVBOKDAYF4UKL4DP23JJM7ZEXG3TJFICUK2MTTM7LPH6GRQA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.121205832 (627h56m13.992288867s) [noise=true , address=145.79.227.49:4360 , piece_id=2HBB4IUFZHBGEAG74CSF3FNZC7GYIKUBRDE64UQLLOX5CBUB34KA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.182495696 (627h56m14.053578741s) [noise=true , address=145.79.227.30:4791 , piece_id=FQNRBDTMLX5DJZBS6ZLA34OAHVZV6UWGZ7WKHRTN524UVMDAPJPA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.229927382 (627h56m14.101010427s) [noise=true , address=145.79.227.33:4897 , piece_id=EL3EGYBRHJ3C7VHL3R72HYROJLDKQDTNBN3DB5BX655PYKUL3PIA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253789819 (627h56m14.124872854s) [noise=true , address=77.90.31.11:4575 , piece_id=XYHKKJKDVQTV6OQKYIILYSI63PJNJY63MHQ4OBN7EW3AC2HTGURA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: <nil> Dec 31 19:25:14.253857889 (627h56m14.124940934s) [seg_pos={0 0}, upload=550237]: Segment reached optimal threshold of 14 pieces. Dec 31 19:25:14.254096944 (627h56m14.125180009s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254106662 (627h56m14.125189707s) [noise=true , address=145.79.227.34:4908 , piece_id=L3ZNJUAUNMKVEHZYUR3XGAEBJYGGESAG7WP3ZZEMGD5NGYUOZOFQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254487577 (627h56m14.125570623s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254497066 (627h56m14.125580111s) [noise=true , address=145.79.227.40:4069 , piece_id=2KR2KFO4F2SWECIZ4WRVLH3VNRP45FS6SRUA7VWUF5KOVZ5IQEEA, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254674845 (627h56m14.1257579s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254682830 (627h56m14.125765865s) [noise=true , address=145.79.227.13:4826 , piece_id=RCOADBJ22JG4PJMO52KCGIISZMEJVCHDDZII55YAOHJMDVIAZAUQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254845420 (627h56m14.125928465s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ): protocol: expected piece hash; context canceled; EOF Dec 31 19:25:14.254854376 (627h56m14.125937421s) [noise=true , address=145.79.227.43:4167 , piece_id=X3DDBXP574QI52UVXVFVRWQA47U2CTWUZOFMXFJXPQ4JF3JUX7MQ, seg_pos={0 0}, upload=550237]: Long tail cancellation due to optimal threshold Dec 31 19:25:14.254875536 (627h56m14.125958581s) [seg_pos={0 0}, upload=550237]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 19:25:14.254958755 (627h56m14.12604181s) [upload=550237]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:25:14.254966259 (627h56m14.126049304s) [upload=550237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:25:14.550705267 (627h56m14.421788312s) [upload=550237]: Done uploading segments... Dec 31 19:25:14.550731055 (627h56m14.42181409s) [upload=550237]: Upload finished. err: <nil> Duration: 27.452877086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X..X.XX....XXX...XX.X.......XXXXXX.......XX..X.X.........] duration: 6.169205659s trace: https://jaeger.dev.storj.io/trace/23d5765d2c5dcc62 FAIL (2025-12-31 19:51:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ) context deadline exceeded ======LOGS====== Dec 31 19:45:35.676293358 (628h16m35.547376403s) [upload=550544]: Starting upload Dec 31 19:45:35.676371106 (628h16m35.547454152s) [upload=550544]: Done starting upload Dec 31 19:45:35.676508036 (628h16m35.547591081s) [upload=550544]: Uploading segments... Dec 31 19:45:35.676768252 (628h16m35.547851287s) [upload=550544]: Got next segment. Inline: false Dec 31 19:45:35.676812156 (628h16m35.547895191s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 19:45:35.676818948 (628h16m35.547901983s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 19:45:35.921497860 (628h16m35.792580906s) [seg_pos={0 0}, upload=550544]: Begin upload segment... Dec 31 19:45:35.921580918 (628h16m35.792663963s) [seg_pos={0 0}, upload=550544]: Done begin upload segment. Dec 31 19:45:35.921608872 (628h16m35.792691917s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921624922 (628h16m35.792707957s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.921704504 (628h16m35.792787539s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922056855 (628h16m35.7931399s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922452558 (628h16m35.793535593s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.922884761 (628h16m35.793967796s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923291886 (628h16m35.794374921s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.923739428 (628h16m35.794822473s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924149439 (628h16m35.795232484s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.924576472 (628h16m35.795659507s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925068259 (628h16m35.796151304s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.925452510 (628h16m35.796535555s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928207797 (628h16m35.799290832s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.928847765 (628h16m35.79993081s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.929387603 (628h16m35.800470648s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:35.930362369 (628h16m35.801445414s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Uploading piece... Dec 31 19:45:37.633748673 (628h16m37.504831708s) [upload=550544]: Next returned nil segment Dec 31 19:45:37.633765546 (628h16m37.504848601s) [upload=550544]: Waiting for error group managing segments... Dec 31 19:45:37.910980199 (628h16m37.782063234s) [noise=true , address=145.79.227.29:4941 , piece_id=AGVGYSTEPUJDEDL2TUHGNBOEPADZVBVRIKSXUKSO2EB6KM5QWJRQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.911343271 (628h16m37.782426316s) [noise=true , address=145.79.227.15:4874 , piece_id=BUNX373EYZMU2YMCTDNLDGV6VJLC3MYT6YDMVMUQTEM56DSUOJRA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.912732806 (628h16m37.783815841s) [noise=true , address=145.79.227.46:4602 , piece_id=O6IDSSOZ3OMBUM56I2C3SIMVBOP54NV4IHHPAA4ASJDFH5CVNOOQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:37.938961860 (628h16m37.810044895s) [noise=true , address=145.79.227.2:4115 , piece_id=NBAOP6RIBXV4RBNIGQV7UNZOL7HGK4KUZZERFEITL2D37CT5DTVQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.003650256 (628h16m37.874733301s) [noise=true , address=145.79.227.45:4229 , piece_id=TQBI7XMUTYDCW6LKHS3YNL52D45ULEP5Y6BZKHQU2PXJQATHPH6A, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.092219332 (628h16m37.963302367s) [noise=true , address=77.90.31.17:4770 , piece_id=ANXHGOY5UD5A3NVOJUBNOWZP72NFQGNAIYTWQH57D2SEUA2YBNNQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.098231691 (628h16m37.969314726s) [noise=true , address=77.90.31.14:4664 , piece_id=TVRQ3SXUHCTQBTQPYA7CHDSVEHZ3IYD7HYB5WL2NYNUN5EXTUPDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.175176875 (628h16m38.04625991s) [noise=true , address=77.90.31.12:4589 , piece_id=CFGB6XHY5HN3ELA2MTMACASGTTFLMHPMLLG2M4FXRFLKEGK3MTBQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.229065616 (628h16m38.100148651s) [noise=true , address=77.90.31.13:4626 , piece_id=M66THCURK4M3OWUHLWVRKYRNHT3KZHAXKMEIGXAGXXLTBPGYJSHQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.246196185 (628h16m38.117279221s) [noise=true , address=45.91.116.12:4225 , piece_id=72DLR4OIJ24RGRBX6WSKWQH7IDIK5HIHOVQNUXEQKD3DCQB2TZMQ, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.253786768 (628h16m38.124869803s) [noise=true , address=45.91.116.6:4976 , piece_id=RLSS5MGT6PSZZBV7QVRDEMK47AMNFH42V4JYTLVKDKGVB4GSERDA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.295960503 (628h16m38.167043538s) [noise=true , address=45.91.116.10:4083 , piece_id=U3NMG2VIEI6MVTZ6SDNHYT2ZIAUFYT6233QYQEAX5U6MMQYWYUEA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296139764 (628h16m38.167222799s) [noise=true , address=77.90.31.11:4555 , piece_id=QIQVYR6CLRAU4WDJIW2INU4E34YYAPBJYG55Q6NJVNAUPR4MN5NA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: <nil> Dec 31 19:45:38.296159883 (628h16m38.167242918s) [seg_pos={0 0}, upload=550544]: Segment reached optimal threshold of 13 pieces. Dec 31 19:45:38.296383608 (628h16m38.167466643s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296390752 (628h16m38.167473787s) [noise=true , address=45.91.116.11:4120 , piece_id=OC2Q3IJUZX54O5GZEYY4R22PZQGZOERXBL7LVG4H4YRVAH36JO4Q, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296565645 (628h16m38.16764868s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:43536->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 19:45:38.296575414 (628h16m38.167658449s) [noise=true , address=77.90.31.10:4517 , piece_id=PPAGZPI3PGYMAO35OKAAJOX35UT35Q6MOAJFQLZGOTAAOG36WOHA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.296986937 (628h16m38.168069972s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 31 19:45:38.296994692 (628h16m38.168077727s) [noise=true , address=77.90.31.16:4736 , piece_id=PNAIOEZ6R5I6FA34XOSLGXY3ZPMJYV2X5UIKETMMD4UYH6NAGYNA, seg_pos={0 0}, upload=550544]: Long tail cancellation due to optimal threshold Dec 31 19:45:38.297009841 (628h16m38.168092876s) [seg_pos={0 0}, upload=550544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 19:45:38.297079423 (628h16m38.168162468s) [upload=550544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 19:45:38.297084222 (628h16m38.168167257s) [upload=550544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 19:45:38.607844643 (628h16m38.478927678s) [upload=550544]: Done uploading segments... Dec 31 19:45:38.607862977 (628h16m38.478946012s) [upload=550544]: Upload finished. err: <nil> Duration: 6.169226227s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)