Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.........X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..] duration: 5.935656933s trace: https://jaeger.dev.storj.io/trace/57e181f2d06ef6d2 FAIL (2025-12-27 15:10:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ) context deadline exceeded ======LOGS====== Dec 27 15:09:05.794789396 (527h40m5.665872431s) [upload=462824]: Starting upload Dec 27 15:09:05.794866463 (527h40m5.665949498s) [upload=462824]: Done starting upload Dec 27 15:09:05.794964570 (527h40m5.666047605s) [upload=462824]: Uploading segments... Dec 27 15:09:05.795115006 (527h40m5.666198041s) [upload=462824]: Got next segment. Inline: false Dec 27 15:09:05.795150353 (527h40m5.666233388s) [upload=462824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:09:05.795156766 (527h40m5.666239811s) [upload=462824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:09:06.037110181 (527h40m5.908193226s) [seg_pos={0 0}, upload=462824]: Begin upload segment... Dec 27 15:09:06.037164534 (527h40m5.908247579s) [seg_pos={0 0}, upload=462824]: Done begin upload segment. Dec 27 15:09:06.037189912 (527h40m5.908272958s) [noise=true , address=145.79.227.4:4185 , piece_id=MMJ6AQZ7MBFDUEA4GHK5P6E5JB4RDKLVR2DE4IMDPZDVIDXZMRDQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.038740705 (527h40m5.90982374s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.039384962 (527h40m5.910468007s) [noise=true , address=145.79.227.36:4518 , piece_id=IRD7OETYB5PT4EVZUWWLNSPHKJKEXVLHUHN5OK6I5TNFMPT7CVTA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.039846671 (527h40m5.910929716s) [noise=true , address=145.79.227.48:4301 , piece_id=NQWPPECU53V2DMQB4QV2AEIIWLEHFUVH6B4Y4LKYKIO5QGSQCVAQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.040398923 (527h40m5.911481998s) [noise=true , address=145.79.227.11:4732 , piece_id=NL7XDVQQ6FPLFKPLAHZ7XA4S6VKN25CMFJFE5MG3UYY446RVX7UQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.041744145 (527h40m5.91282719s) [noise=true , address=45.91.116.14:4191 , piece_id=7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.042206174 (527h40m5.913289209s) [noise=true , address=145.79.227.35:4483 , piece_id=PMETF743YGP5EWCA54XSJY4M53AGISOEWZIIAB6BHQCTCOQNP77Q, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.042983955 (527h40m5.91406699s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.043700279 (527h40m5.914783324s) [noise=true , address=145.79.227.5:4221 , piece_id=SKOIWIPXBUQEXC6PMXMELDD42NCN3RLYE7UICL72HR5CBLRFPCMA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.046511292 (527h40m5.917594337s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.048481563 (527h40m5.919564608s) [noise=true , address=145.79.227.46:4603 , piece_id=FQRXJ3P7GU5SHFOFNOQXQU7IMFOBOPDSTIPXZRHOCK4EPFX7IBUA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.050586893 (527h40m5.921669928s) [noise=true , address=145.79.227.3:4151 , piece_id=F5WOSQTRLDPCGNURPQ6EJYNEGS2ELKNNM32AF4ORUKQ5S34TYB7Q, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.052974288 (527h40m5.924057323s) [noise=true , address=77.90.31.14:4664 , piece_id=ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.053860085 (527h40m5.92494312s) [noise=true , address=145.79.227.39:4015 , piece_id=QPRKPNQOSWNECGBVW3J5U3LW7IBMZSTRI6CWSB7YS6DEPGIIXCRA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.055784339 (527h40m5.926867384s) [noise=true , address=45.91.116.11:4117 , piece_id=A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.058940479 (527h40m5.930023515s) [noise=true , address=145.79.227.28:4726 , piece_id=VB4MAZ2ZAQSBOKTWYLXFCKMS6UPBCQMG6UD5CCQ2EI7NCAV7VBJQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:07.771249072 (527h40m7.642332118s) [upload=462824]: Next returned nil segment Dec 27 15:09:07.771270863 (527h40m7.642353908s) [upload=462824]: Waiting for error group managing segments... Dec 27 15:09:08.047103661 (527h40m7.918186706s) [noise=true , address=145.79.227.48:4301 , piece_id=NQWPPECU53V2DMQB4QV2AEIIWLEHFUVH6B4Y4LKYKIO5QGSQCVAQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.051537523 (527h40m7.922620558s) [noise=true , address=145.79.227.35:4483 , piece_id=PMETF743YGP5EWCA54XSJY4M53AGISOEWZIIAB6BHQCTCOQNP77Q, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.057900229 (527h40m7.928983264s) [noise=true , address=145.79.227.11:4732 , piece_id=NL7XDVQQ6FPLFKPLAHZ7XA4S6VKN25CMFJFE5MG3UYY446RVX7UQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.058250626 (527h40m7.929333671s) [noise=true , address=145.79.227.5:4221 , piece_id=SKOIWIPXBUQEXC6PMXMELDD42NCN3RLYE7UICL72HR5CBLRFPCMA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.064404244 (527h40m7.935487279s) [noise=true , address=145.79.227.36:4518 , piece_id=IRD7OETYB5PT4EVZUWWLNSPHKJKEXVLHUHN5OK6I5TNFMPT7CVTA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.066479396 (527h40m7.937562431s) [noise=true , address=145.79.227.3:4151 , piece_id=F5WOSQTRLDPCGNURPQ6EJYNEGS2ELKNNM32AF4ORUKQ5S34TYB7Q, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.069637559 (527h40m7.940720594s) [noise=true , address=145.79.227.46:4603 , piece_id=FQRXJ3P7GU5SHFOFNOQXQU7IMFOBOPDSTIPXZRHOCK4EPFX7IBUA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.082490464 (527h40m7.953573509s) [noise=true , address=145.79.227.4:4185 , piece_id=MMJ6AQZ7MBFDUEA4GHK5P6E5JB4RDKLVR2DE4IMDPZDVIDXZMRDQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.085012386 (527h40m7.956095421s) [noise=true , address=145.79.227.39:4015 , piece_id=QPRKPNQOSWNECGBVW3J5U3LW7IBMZSTRI6CWSB7YS6DEPGIIXCRA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.104923540 (527h40m7.976006585s) [noise=true , address=145.79.227.28:4726 , piece_id=VB4MAZ2ZAQSBOKTWYLXFCKMS6UPBCQMG6UD5CCQ2EI7NCAV7VBJQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.259639268 (527h40m8.130722303s) [noise=true , address=77.90.31.14:4664 , piece_id=ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.281183151 (527h40m8.152266186s) [noise=true , address=45.91.116.11:4117 , piece_id=A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.323734046 (527h40m8.194817091s) [noise=true , address=45.91.116.14:4191 , piece_id=7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.324304783 (527h40m8.195387828s) [seg_pos={0 0}, upload=462824]: Segment reached optimal threshold of 13 pieces. Dec 27 15:09:08.324835925 (527h40m8.19591896s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 15:09:08.324844040 (527h40m8.195927075s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.325669883 (527h40m8.196752918s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 15:09:08.325842582 (527h40m8.196925627s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.326213066 (527h40m8.197296101s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection Dec 27 15:09:08.326230440 (527h40m8.197313475s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.326862343 (527h40m8.197945388s) [seg_pos={0 0}, upload=462824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:09:08.327130364 (527h40m8.198213409s) [upload=462824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:09:08.327141805 (527h40m8.19822485s) [upload=462824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:09:08.613521966 (527h40m8.484605011s) [upload=462824]: Done uploading segments... Dec 27 15:09:08.613547945 (527h40m8.48463098s) [upload=462824]: Upload finished. err: <nil> Duration: 5.935697709s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.565230179s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:10:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 7.565260758s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X..X...........X..............X....X............] duration: 40.638849845s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:09:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 40.638876806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX] duration: 5.001473392s trace: https://jaeger.dev.storj.io/trace/6b76f8dc74ac4b56 FAIL (2025-12-27 15:09:50 UTC) orbiter warning: context deadline exceeded Duration: 5.001498989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.........X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX.] duration: 5.852411475s trace: https://jaeger.dev.storj.io/trace/57e181f2d06ef6d2 FAIL (2025-12-27 15:09:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ) context deadline exceeded ======LOGS====== Dec 27 15:09:05.794789396 (527h40m5.665872431s) [upload=462824]: Starting upload Dec 27 15:09:05.794866463 (527h40m5.665949498s) [upload=462824]: Done starting upload Dec 27 15:09:05.794964570 (527h40m5.666047605s) [upload=462824]: Uploading segments... Dec 27 15:09:05.795115006 (527h40m5.666198041s) [upload=462824]: Got next segment. Inline: false Dec 27 15:09:05.795150353 (527h40m5.666233388s) [upload=462824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:09:05.795156766 (527h40m5.666239811s) [upload=462824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:09:06.037110181 (527h40m5.908193226s) [seg_pos={0 0}, upload=462824]: Begin upload segment... Dec 27 15:09:06.037164534 (527h40m5.908247579s) [seg_pos={0 0}, upload=462824]: Done begin upload segment. Dec 27 15:09:06.037189912 (527h40m5.908272958s) [noise=true , address=145.79.227.4:4185 , piece_id=MMJ6AQZ7MBFDUEA4GHK5P6E5JB4RDKLVR2DE4IMDPZDVIDXZMRDQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.038740705 (527h40m5.90982374s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.039384962 (527h40m5.910468007s) [noise=true , address=145.79.227.36:4518 , piece_id=IRD7OETYB5PT4EVZUWWLNSPHKJKEXVLHUHN5OK6I5TNFMPT7CVTA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.039846671 (527h40m5.910929716s) [noise=true , address=145.79.227.48:4301 , piece_id=NQWPPECU53V2DMQB4QV2AEIIWLEHFUVH6B4Y4LKYKIO5QGSQCVAQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.040398923 (527h40m5.911481998s) [noise=true , address=145.79.227.11:4732 , piece_id=NL7XDVQQ6FPLFKPLAHZ7XA4S6VKN25CMFJFE5MG3UYY446RVX7UQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.041744145 (527h40m5.91282719s) [noise=true , address=45.91.116.14:4191 , piece_id=7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.042206174 (527h40m5.913289209s) [noise=true , address=145.79.227.35:4483 , piece_id=PMETF743YGP5EWCA54XSJY4M53AGISOEWZIIAB6BHQCTCOQNP77Q, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.042983955 (527h40m5.91406699s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.043700279 (527h40m5.914783324s) [noise=true , address=145.79.227.5:4221 , piece_id=SKOIWIPXBUQEXC6PMXMELDD42NCN3RLYE7UICL72HR5CBLRFPCMA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.046511292 (527h40m5.917594337s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.048481563 (527h40m5.919564608s) [noise=true , address=145.79.227.46:4603 , piece_id=FQRXJ3P7GU5SHFOFNOQXQU7IMFOBOPDSTIPXZRHOCK4EPFX7IBUA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.050586893 (527h40m5.921669928s) [noise=true , address=145.79.227.3:4151 , piece_id=F5WOSQTRLDPCGNURPQ6EJYNEGS2ELKNNM32AF4ORUKQ5S34TYB7Q, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.052974288 (527h40m5.924057323s) [noise=true , address=77.90.31.14:4664 , piece_id=ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.053860085 (527h40m5.92494312s) [noise=true , address=145.79.227.39:4015 , piece_id=QPRKPNQOSWNECGBVW3J5U3LW7IBMZSTRI6CWSB7YS6DEPGIIXCRA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.055784339 (527h40m5.926867384s) [noise=true , address=45.91.116.11:4117 , piece_id=A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.058940479 (527h40m5.930023515s) [noise=true , address=145.79.227.28:4726 , piece_id=VB4MAZ2ZAQSBOKTWYLXFCKMS6UPBCQMG6UD5CCQ2EI7NCAV7VBJQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:07.771249072 (527h40m7.642332118s) [upload=462824]: Next returned nil segment Dec 27 15:09:07.771270863 (527h40m7.642353908s) [upload=462824]: Waiting for error group managing segments... Dec 27 15:09:08.047103661 (527h40m7.918186706s) [noise=true , address=145.79.227.48:4301 , piece_id=NQWPPECU53V2DMQB4QV2AEIIWLEHFUVH6B4Y4LKYKIO5QGSQCVAQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.051537523 (527h40m7.922620558s) [noise=true , address=145.79.227.35:4483 , piece_id=PMETF743YGP5EWCA54XSJY4M53AGISOEWZIIAB6BHQCTCOQNP77Q, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.057900229 (527h40m7.928983264s) [noise=true , address=145.79.227.11:4732 , piece_id=NL7XDVQQ6FPLFKPLAHZ7XA4S6VKN25CMFJFE5MG3UYY446RVX7UQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.058250626 (527h40m7.929333671s) [noise=true , address=145.79.227.5:4221 , piece_id=SKOIWIPXBUQEXC6PMXMELDD42NCN3RLYE7UICL72HR5CBLRFPCMA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.064404244 (527h40m7.935487279s) [noise=true , address=145.79.227.36:4518 , piece_id=IRD7OETYB5PT4EVZUWWLNSPHKJKEXVLHUHN5OK6I5TNFMPT7CVTA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.066479396 (527h40m7.937562431s) [noise=true , address=145.79.227.3:4151 , piece_id=F5WOSQTRLDPCGNURPQ6EJYNEGS2ELKNNM32AF4ORUKQ5S34TYB7Q, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.069637559 (527h40m7.940720594s) [noise=true , address=145.79.227.46:4603 , piece_id=FQRXJ3P7GU5SHFOFNOQXQU7IMFOBOPDSTIPXZRHOCK4EPFX7IBUA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.082490464 (527h40m7.953573509s) [noise=true , address=145.79.227.4:4185 , piece_id=MMJ6AQZ7MBFDUEA4GHK5P6E5JB4RDKLVR2DE4IMDPZDVIDXZMRDQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.085012386 (527h40m7.956095421s) [noise=true , address=145.79.227.39:4015 , piece_id=QPRKPNQOSWNECGBVW3J5U3LW7IBMZSTRI6CWSB7YS6DEPGIIXCRA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.104923540 (527h40m7.976006585s) [noise=true , address=145.79.227.28:4726 , piece_id=VB4MAZ2ZAQSBOKTWYLXFCKMS6UPBCQMG6UD5CCQ2EI7NCAV7VBJQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.259639268 (527h40m8.130722303s) [noise=true , address=77.90.31.14:4664 , piece_id=ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.281183151 (527h40m8.152266186s) [noise=true , address=45.91.116.11:4117 , piece_id=A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.323734046 (527h40m8.194817091s) [noise=true , address=45.91.116.14:4191 , piece_id=7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.324304783 (527h40m8.195387828s) [seg_pos={0 0}, upload=462824]: Segment reached optimal threshold of 13 pieces. Dec 27 15:09:08.324835925 (527h40m8.19591896s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 15:09:08.324844040 (527h40m8.195927075s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.325669883 (527h40m8.196752918s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 15:09:08.325842582 (527h40m8.196925627s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.326213066 (527h40m8.197296101s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection Dec 27 15:09:08.326230440 (527h40m8.197313475s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.326862343 (527h40m8.197945388s) [seg_pos={0 0}, upload=462824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:09:08.327130364 (527h40m8.198213409s) [upload=462824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:09:08.327141805 (527h40m8.19822485s) [upload=462824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:09:08.613521966 (527h40m8.484605011s) [upload=462824]: Done uploading segments... Dec 27 15:09:08.613547945 (527h40m8.48463098s) [upload=462824]: Upload finished. err: <nil> Duration: 5.85243492s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 29.01567189s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:09:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 29.015702458s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.........X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX] duration: 30.003355383s trace: https://jaeger.dev.storj.io/trace/57e181f2d06ef6d2 FAIL (2025-12-27 15:09:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ) context deadline exceeded ======LOGS====== Dec 27 15:09:05.794789396 (527h40m5.665872431s) [upload=462824]: Starting upload Dec 27 15:09:05.794866463 (527h40m5.665949498s) [upload=462824]: Done starting upload Dec 27 15:09:05.794964570 (527h40m5.666047605s) [upload=462824]: Uploading segments... Dec 27 15:09:05.795115006 (527h40m5.666198041s) [upload=462824]: Got next segment. Inline: false Dec 27 15:09:05.795150353 (527h40m5.666233388s) [upload=462824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:09:05.795156766 (527h40m5.666239811s) [upload=462824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:09:06.037110181 (527h40m5.908193226s) [seg_pos={0 0}, upload=462824]: Begin upload segment... Dec 27 15:09:06.037164534 (527h40m5.908247579s) [seg_pos={0 0}, upload=462824]: Done begin upload segment. Dec 27 15:09:06.037189912 (527h40m5.908272958s) [noise=true , address=145.79.227.4:4185 , piece_id=MMJ6AQZ7MBFDUEA4GHK5P6E5JB4RDKLVR2DE4IMDPZDVIDXZMRDQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.038740705 (527h40m5.90982374s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.039384962 (527h40m5.910468007s) [noise=true , address=145.79.227.36:4518 , piece_id=IRD7OETYB5PT4EVZUWWLNSPHKJKEXVLHUHN5OK6I5TNFMPT7CVTA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.039846671 (527h40m5.910929716s) [noise=true , address=145.79.227.48:4301 , piece_id=NQWPPECU53V2DMQB4QV2AEIIWLEHFUVH6B4Y4LKYKIO5QGSQCVAQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.040398923 (527h40m5.911481998s) [noise=true , address=145.79.227.11:4732 , piece_id=NL7XDVQQ6FPLFKPLAHZ7XA4S6VKN25CMFJFE5MG3UYY446RVX7UQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.041744145 (527h40m5.91282719s) [noise=true , address=45.91.116.14:4191 , piece_id=7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.042206174 (527h40m5.913289209s) [noise=true , address=145.79.227.35:4483 , piece_id=PMETF743YGP5EWCA54XSJY4M53AGISOEWZIIAB6BHQCTCOQNP77Q, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.042983955 (527h40m5.91406699s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.043700279 (527h40m5.914783324s) [noise=true , address=145.79.227.5:4221 , piece_id=SKOIWIPXBUQEXC6PMXMELDD42NCN3RLYE7UICL72HR5CBLRFPCMA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.046511292 (527h40m5.917594337s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.048481563 (527h40m5.919564608s) [noise=true , address=145.79.227.46:4603 , piece_id=FQRXJ3P7GU5SHFOFNOQXQU7IMFOBOPDSTIPXZRHOCK4EPFX7IBUA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.050586893 (527h40m5.921669928s) [noise=true , address=145.79.227.3:4151 , piece_id=F5WOSQTRLDPCGNURPQ6EJYNEGS2ELKNNM32AF4ORUKQ5S34TYB7Q, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.052974288 (527h40m5.924057323s) [noise=true , address=77.90.31.14:4664 , piece_id=ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.053860085 (527h40m5.92494312s) [noise=true , address=145.79.227.39:4015 , piece_id=QPRKPNQOSWNECGBVW3J5U3LW7IBMZSTRI6CWSB7YS6DEPGIIXCRA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.055784339 (527h40m5.926867384s) [noise=true , address=45.91.116.11:4117 , piece_id=A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:06.058940479 (527h40m5.930023515s) [noise=true , address=145.79.227.28:4726 , piece_id=VB4MAZ2ZAQSBOKTWYLXFCKMS6UPBCQMG6UD5CCQ2EI7NCAV7VBJQ, seg_pos={0 0}, upload=462824]: Uploading piece... Dec 27 15:09:07.771249072 (527h40m7.642332118s) [upload=462824]: Next returned nil segment Dec 27 15:09:07.771270863 (527h40m7.642353908s) [upload=462824]: Waiting for error group managing segments... Dec 27 15:09:08.047103661 (527h40m7.918186706s) [noise=true , address=145.79.227.48:4301 , piece_id=NQWPPECU53V2DMQB4QV2AEIIWLEHFUVH6B4Y4LKYKIO5QGSQCVAQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.051537523 (527h40m7.922620558s) [noise=true , address=145.79.227.35:4483 , piece_id=PMETF743YGP5EWCA54XSJY4M53AGISOEWZIIAB6BHQCTCOQNP77Q, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.057900229 (527h40m7.928983264s) [noise=true , address=145.79.227.11:4732 , piece_id=NL7XDVQQ6FPLFKPLAHZ7XA4S6VKN25CMFJFE5MG3UYY446RVX7UQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.058250626 (527h40m7.929333671s) [noise=true , address=145.79.227.5:4221 , piece_id=SKOIWIPXBUQEXC6PMXMELDD42NCN3RLYE7UICL72HR5CBLRFPCMA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.064404244 (527h40m7.935487279s) [noise=true , address=145.79.227.36:4518 , piece_id=IRD7OETYB5PT4EVZUWWLNSPHKJKEXVLHUHN5OK6I5TNFMPT7CVTA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.066479396 (527h40m7.937562431s) [noise=true , address=145.79.227.3:4151 , piece_id=F5WOSQTRLDPCGNURPQ6EJYNEGS2ELKNNM32AF4ORUKQ5S34TYB7Q, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.069637559 (527h40m7.940720594s) [noise=true , address=145.79.227.46:4603 , piece_id=FQRXJ3P7GU5SHFOFNOQXQU7IMFOBOPDSTIPXZRHOCK4EPFX7IBUA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.082490464 (527h40m7.953573509s) [noise=true , address=145.79.227.4:4185 , piece_id=MMJ6AQZ7MBFDUEA4GHK5P6E5JB4RDKLVR2DE4IMDPZDVIDXZMRDQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.085012386 (527h40m7.956095421s) [noise=true , address=145.79.227.39:4015 , piece_id=QPRKPNQOSWNECGBVW3J5U3LW7IBMZSTRI6CWSB7YS6DEPGIIXCRA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.104923540 (527h40m7.976006585s) [noise=true , address=145.79.227.28:4726 , piece_id=VB4MAZ2ZAQSBOKTWYLXFCKMS6UPBCQMG6UD5CCQ2EI7NCAV7VBJQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.259639268 (527h40m8.130722303s) [noise=true , address=77.90.31.14:4664 , piece_id=ZRP3UZWBH655T7J4IEWUYOF2Z3L3JWBHBAOGM2EYGGRA7N6ZJ6RQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.281183151 (527h40m8.152266186s) [noise=true , address=45.91.116.11:4117 , piece_id=A6PYT2Y5IEHD3QVIOSSEARH6OSLIFT6AIUXVNUMHU3W552XVB5TA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.323734046 (527h40m8.194817091s) [noise=true , address=45.91.116.14:4191 , piece_id=7MC4H76CQSGWMKR25746KA3VWJN3EPPKJCA42ST5KWWTJLP4HIAQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: <nil> Dec 27 15:09:08.324304783 (527h40m8.195387828s) [seg_pos={0 0}, upload=462824]: Segment reached optimal threshold of 13 pieces. Dec 27 15:09:08.324835925 (527h40m8.19591896s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 15:09:08.324844040 (527h40m8.195927075s) [noise=true , address=45.91.116.4:4939 , piece_id=VNSLKTFWOWILI3VWZQUU4AG3NJHGVPUZHLQIYQEHC2FM3VEKLGFQ, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.325669883 (527h40m8.196752918s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 15:09:08.325842582 (527h40m8.196925627s) [noise=true , address=45.91.116.7:4010 , piece_id=E2JBENSKSHF3O2NT5W5WML4TEDAKY7EHZQMR2STHBQJPNPVBL2OA, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.326213066 (527h40m8.197296101s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->45.91.116.13:4155: use of closed network connection Dec 27 15:09:08.326230440 (527h40m8.197313475s) [noise=true , address=45.91.116.13:4155 , piece_id=64PUR4EZ6IVWR3EVDKBJLAZJUG2ADZAHEE62CBLZZM4JYLRZ76DQ, seg_pos={0 0}, upload=462824]: Long tail cancellation due to optimal threshold Dec 27 15:09:08.326862343 (527h40m8.197945388s) [seg_pos={0 0}, upload=462824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:09:08.327130364 (527h40m8.198213409s) [upload=462824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:09:08.327141805 (527h40m8.19822485s) [upload=462824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:09:08.613521966 (527h40m8.484605011s) [upload=462824]: Done uploading segments... Dec 27 15:09:08.613547945 (527h40m8.48463098s) [upload=462824]: Upload finished. err: <nil> Duration: 30.003370843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.........X.X....X...X.....X..XX.......X..X.X..X...XXXXXX] duration: 30.003268401s trace: https://jaeger.dev.storj.io/trace/34225bc5d77c8bd2 FAIL (2025-12-27 15:09:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: H6VS5VPFRFYI3IAZZR643RIINOAETDPI7BVADKYUNYW5UWNXJJHA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: OR5TDKVIZSKCLEER4VAO6TF4DXEJCUOQXZAB5Q2LFYQTZUOQSRCA) context deadline exceeded ======LOGS====== Dec 27 15:08:35.563006792 (527h39m35.434089837s) [upload=462815]: Starting upload Dec 27 15:08:35.563146759 (527h39m35.434229814s) [upload=462815]: Done starting upload Dec 27 15:08:35.563608307 (527h39m35.434691353s) [upload=462815]: Uploading segments... Dec 27 15:08:35.563825953 (527h39m35.434909008s) [upload=462815]: Got next segment. Inline: false Dec 27 15:08:35.563897699 (527h39m35.434980734s) [upload=462815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:08:35.563903991 (527h39m35.434987036s) [upload=462815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:08:35.813754613 (527h39m35.684837678s) [seg_pos={0 0}, upload=462815]: Begin upload segment... Dec 27 15:08:35.813858351 (527h39m35.684941396s) [seg_pos={0 0}, upload=462815]: Done begin upload segment. Dec 27 15:08:35.813911993 (527h39m35.684995029s) [noise=true , address=45.91.116.10:4083 , piece_id=MS2QGCBU4UTTZUPBV3FLXQJA6UZEG4QUID2XRRVFZG4GQBGSXEJA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.813934978 (527h39m35.685018013s) [noise=true , address=145.79.227.12:4767 , piece_id=N23ZAI75X5OKPINIRCC5H77QE3FHVCN5R4XMXGICHROMZZRO45VA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.814485505 (527h39m35.68556854s) [noise=true , address=145.79.227.13:4804 , piece_id=DC3S2ERCTWSHS2R2QAHSXQI47VDUNSRUF3QQORDMZJE23MQK7X5Q, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.814896257 (527h39m35.685979302s) [noise=true , address=145.79.227.18:4007 , piece_id=WKLPWJZZXW23GU2J74MKOLHMMAZ5ZBUHX6YCTWL3TPUPHJR3CNOA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.815452957 (527h39m35.686536003s) [noise=true , address=77.90.31.14:4661 , piece_id=54WGLGZX53PH3TF67AGEMZIURXILWXZSAB6QH2MXNJYIEBQMWQLA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.815525936 (527h39m35.686608971s) [noise=true , address=145.79.227.37:4567 , piece_id=YDL7QJK7R6KS3L7K2OZM5VPYHM44BPKKYYUHJURHK3GC2HLHNK2A, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.816089259 (527h39m35.687172304s) [noise=true , address=77.90.31.11:4553 , piece_id=QF6MFVRPY25FCHSQ5KN4V7FI4SNZDEW4QS6MCB7GJW7IKV2NBJEQ, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.816213285 (527h39m35.687296331s) [noise=true , address=145.79.227.29:4943 , piece_id=NA5OM7OJZKHJUFES5XGCOGPHD4RVRYDZVMHCGYEB54PTJYIK45IA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.816657041 (527h39m35.687740086s) [noise=true , address=145.79.227.42:4121 , piece_id=ZNONRAIMURTYRBKZEYRYC4BLSKJGOJLRLAL3WWE6DBMVA2EMASEA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.816800434 (527h39m35.687883479s) [noise=true , address=145.79.227.17:4971 , piece_id=PTN2SULD34PP5EPTX2FMSRTA4F3GHJ5AZ4F4XSVONK6LBLK5TWLA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.817318460 (527h39m35.688401495s) [noise=true , address=45.91.116.13:4153 , piece_id=7OHGMTCNNY6PJWHOE6HKBJD3XI7PQKOQ5PKJIXJI3YDGYGFFGQVA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.817384947 (527h39m35.688467992s) [noise=true , address=45.91.116.5:4300 , piece_id=W5QDCBB57WUFDXBGUN2ZRN2KVTMS62R5MMHFS2IMX5KXWVTXZUXQ, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.817930034 (527h39m35.689013069s) [noise=true , address=77.90.31.15:4700 , piece_id=H6VS5VPFRFYI3IAZZR643RIINOAETDPI7BVADKYUNYW5UWNXJJHA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.818000418 (527h39m35.689083463s) [noise=true , address=145.79.227.40:4051 , piece_id=CGBMB4XKMQPPPRJJEEFUZFYLHE4WP7X5GSYXS4SPSS4G34ENH4BA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.818654384 (527h39m35.689737429s) [noise=true , address=45.91.116.11:4117 , piece_id=OR5TDKVIZSKCLEER4VAO6TF4DXEJCUOQXZAB5Q2LFYQTZUOQSRCA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:35.820074458 (527h39m35.691157493s) [noise=true , address=145.79.227.8:4329 , piece_id=TKA4M4GAONR6IX3TE3SAJJQHRB64PMYYNWUOJHI3ZT4G7LSGAXMA, seg_pos={0 0}, upload=462815]: Uploading piece... Dec 27 15:08:37.538367748 (527h39m37.409450793s) [upload=462815]: Next returned nil segment Dec 27 15:08:37.538390491 (527h39m37.409473536s) [upload=462815]: Waiting for error group managing segments... Dec 27 15:08:37.820890293 (527h39m37.691973328s) [noise=true , address=145.79.227.12:4767 , piece_id=N23ZAI75X5OKPINIRCC5H77QE3FHVCN5R4XMXGICHROMZZRO45VA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:37.822572096 (527h39m37.693655131s) [noise=true , address=145.79.227.18:4007 , piece_id=WKLPWJZZXW23GU2J74MKOLHMMAZ5ZBUHX6YCTWL3TPUPHJR3CNOA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:37.827839576 (527h39m37.698922611s) [noise=true , address=145.79.227.42:4121 , piece_id=ZNONRAIMURTYRBKZEYRYC4BLSKJGOJLRLAL3WWE6DBMVA2EMASEA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:37.828122564 (527h39m37.699205599s) [noise=true , address=145.79.227.13:4804 , piece_id=DC3S2ERCTWSHS2R2QAHSXQI47VDUNSRUF3QQORDMZJE23MQK7X5Q, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:37.834546207 (527h39m37.705629242s) [noise=true , address=145.79.227.29:4943 , piece_id=NA5OM7OJZKHJUFES5XGCOGPHD4RVRYDZVMHCGYEB54PTJYIK45IA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:37.845963858 (527h39m37.717046893s) [noise=true , address=145.79.227.37:4567 , piece_id=YDL7QJK7R6KS3L7K2OZM5VPYHM44BPKKYYUHJURHK3GC2HLHNK2A, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:37.845981340 (527h39m37.717064386s) [noise=true , address=145.79.227.17:4971 , piece_id=PTN2SULD34PP5EPTX2FMSRTA4F3GHJ5AZ4F4XSVONK6LBLK5TWLA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:37.846461796 (527h39m37.717544831s) [noise=true , address=145.79.227.8:4329 , piece_id=TKA4M4GAONR6IX3TE3SAJJQHRB64PMYYNWUOJHI3ZT4G7LSGAXMA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:38.055386086 (527h39m37.926469121s) [noise=true , address=45.91.116.5:4300 , piece_id=W5QDCBB57WUFDXBGUN2ZRN2KVTMS62R5MMHFS2IMX5KXWVTXZUXQ, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:38.182898493 (527h39m38.053981528s) [noise=true , address=45.91.116.10:4083 , piece_id=MS2QGCBU4UTTZUPBV3FLXQJA6UZEG4QUID2XRRVFZG4GQBGSXEJA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:38.337603170 (527h39m38.208686225s) [noise=true , address=45.91.116.11:4117 , piece_id=OR5TDKVIZSKCLEER4VAO6TF4DXEJCUOQXZAB5Q2LFYQTZUOQSRCA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:38.796650721 (527h39m38.667733756s) [noise=true , address=77.90.31.11:4553 , piece_id=QF6MFVRPY25FCHSQ5KN4V7FI4SNZDEW4QS6MCB7GJW7IKV2NBJEQ, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:38.970765338 (527h39m38.841848383s) [noise=true , address=77.90.31.15:4700 , piece_id=H6VS5VPFRFYI3IAZZR643RIINOAETDPI7BVADKYUNYW5UWNXJJHA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: <nil> Dec 27 15:08:38.970788302 (527h39m38.841871347s) [seg_pos={0 0}, upload=462815]: Segment reached optimal threshold of 13 pieces. Dec 27 15:08:38.971000416 (527h39m38.842083451s) [noise=true , address=145.79.227.40:4051 , piece_id=CGBMB4XKMQPPPRJJEEFUZFYLHE4WP7X5GSYXS4SPSS4G34ENH4BA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 15:08:38.971008942 (527h39m38.842091977s) [noise=true , address=145.79.227.40:4051 , piece_id=CGBMB4XKMQPPPRJJEEFUZFYLHE4WP7X5GSYXS4SPSS4G34ENH4BA, seg_pos={0 0}, upload=462815]: Long tail cancellation due to optimal threshold Dec 27 15:08:38.971191479 (527h39m38.842274524s) [noise=true , address=77.90.31.14:4661 , piece_id=54WGLGZX53PH3TF67AGEMZIURXILWXZSAB6QH2MXNJYIEBQMWQLA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 15:08:38.971210836 (527h39m38.842293872s) [noise=true , address=77.90.31.14:4661 , piece_id=54WGLGZX53PH3TF67AGEMZIURXILWXZSAB6QH2MXNJYIEBQMWQLA, seg_pos={0 0}, upload=462815]: Long tail cancellation due to optimal threshold Dec 27 15:08:38.971320736 (527h39m38.842403791s) [noise=true , address=45.91.116.13:4153 , piece_id=7OHGMTCNNY6PJWHOE6HKBJD3XI7PQKOQ5PKJIXJI3YDGYGFFGQVA, seg_pos={0 0}, upload=462815]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 15:08:38.971330734 (527h39m38.842413769s) [noise=true , address=45.91.116.13:4153 , piece_id=7OHGMTCNNY6PJWHOE6HKBJD3XI7PQKOQ5PKJIXJI3YDGYGFFGQVA, seg_pos={0 0}, upload=462815]: Long tail cancellation due to optimal threshold Dec 27 15:08:38.971375991 (527h39m38.842459026s) [seg_pos={0 0}, upload=462815]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:08:38.971480460 (527h39m38.842563505s) [upload=462815]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:08:38.971488736 (527h39m38.842571771s) [upload=462815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:08:39.270158057 (527h39m39.141241092s) [upload=462815]: Done uploading segments... Dec 27 15:08:39.270178125 (527h39m39.14126116s) [upload=462815]: Upload finished. err: <nil> Duration: 30.003281777s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 13.690382018s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:08:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 13.690426051s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXX] duration: 5.001131682s trace: https://jaeger.dev.storj.io/trace/286790827839fe7e FAIL (2025-12-27 15:08:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00114053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X.........X.X....X...X.....X..XX.......X..X.X..X...XXXXX] duration: 30.002967221s trace: https://jaeger.dev.storj.io/trace/5a3ad545c8fa0869 FAIL (2025-12-27 15:08:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: XBXENLARRGNHVUUTD5ILUE6LTIZDE77BCFNHXHPEZTY4CGH7254Q) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 4WZOXQACUODLZ4PU65TVUMTEMDJ6VABH7UTZBY555IQOFPX5NKMQ) context deadline exceeded ======LOGS====== upload=462809]: Got next segment. Inline: false Dec 27 15:08:05.329989454 (527h39m5.201072489s) [upload=462809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:08:05.329994012 (527h39m5.201077047s) [upload=462809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:08:05.578728037 (527h39m5.449811082s) [seg_pos={0 0}, upload=462809]: Begin upload segment... Dec 27 15:08:05.578796067 (527h39m5.449879112s) [seg_pos={0 0}, upload=462809]: Done begin upload segment. Dec 27 15:08:05.578903021 (527h39m5.449986066s) [noise=true , address=45.91.116.11:4120 , piece_id=HV23MQWNDNI2QEBVR6JPPMP22PRGIBOEGGWFLUSQT5SPAPPRTANQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.578944881 (527h39m5.450027926s) [noise=true , address=145.79.227.8:4329 , piece_id=GMOGZCIZZ32562BOISN6JI2QQKXCRDDYHSGLTT6KFELSIMQAOF2Q, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.579882176 (527h39m5.450965221s) [noise=true , address=145.79.227.20:4438 , piece_id=4Q5AD4KN5HFTAXNMMKSF6N7X66YWLSWFNEFZHPQQ5YP2IFHF7PIQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.580690685 (527h39m5.45177373s) [noise=true , address=77.90.31.19:4841 , piece_id=QM3DYB5BHN3PEWRD4WRWGP77XPMHI54DIKYBQAXTFJIRL3VUDR6A, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.581553639 (527h39m5.452636684s) [noise=true , address=145.79.227.7:4294 , piece_id=NI5ICWVHL2VNC3UTJRMVRC3BUGOFDQ6W3AXURW2X53G4S3C426YQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.581852337 (527h39m5.452935382s) [noise=true , address=145.79.227.3:4151 , piece_id=CMTD3ORK3LXXIZUNOWUUYQHLW3I6FX3MTYXVNUHUDYYK2ZCJTE6Q, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.582466647 (527h39m5.453549682s) [noise=true , address=145.79.227.45:4230 , piece_id=BBZ4NU6EFFH2G56TPZUCDYX4AEECF2O24MUPOHAXTC7L5ZJSNKYA, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.582495562 (527h39m5.453578597s) [noise=true , address=145.79.227.52:4445 , piece_id=DZN75CIDAUOWAAERSBJV3SMTBGDJBWFRJRIVL3NSJXQOSBPNPQTQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.583045029 (527h39m5.454128074s) [noise=true , address=45.91.116.6:4975 , piece_id=JSX4WITHXMMUFT7UNI6M3B6UD6LLUUQWNICAHZFDY6AOGAYLWRKQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.583415443 (527h39m5.454498488s) [noise=true , address=45.91.116.14:4192 , piece_id=XGK2R4EJGAVV7XH7TY43W4XM3KSM7VMNJROAZLWPZB6Z2MQL4Y6Q, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.584105397 (527h39m5.455188432s) [noise=true , address=145.79.227.37:4565 , piece_id=3N3XYM576355JXLEJI75QH4IQ57RMEDE4FVTEGLAJD3QMKKHQ3HQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.584565573 (527h39m5.455648608s) [noise=true , address=145.79.227.23:4546 , piece_id=3HECX7NWRC2YDT4AOOGWBKD4WGJSCXJ2ETQIPRVCSOCJBFVOEZ6A, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.585157561 (527h39m5.456240606s) [noise=true , address=45.91.116.4:4937 , piece_id=4LHNTITWH4GVYH4FCR4377UMVXG4S6B37E3W7DQJJEYWWYMAPSTQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.585250798 (527h39m5.456333844s) [noise=true , address=77.90.31.14:4661 , piece_id=XBXENLARRGNHVUUTD5ILUE6LTIZDE77BCFNHXHPEZTY4CGH7254Q, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.586265661 (527h39m5.457348706s) [noise=true , address=45.91.116.10:4083 , piece_id=4WZOXQACUODLZ4PU65TVUMTEMDJ6VABH7UTZBY555IQOFPX5NKMQ, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:05.586576252 (527h39m5.457659287s) [noise=true , address=145.79.227.29:4943 , piece_id=HMOJ4YN6V37Y73FJZ53M5SSRO3ADTQPVVFF7Y7RVCZ3SIODCV3BA, seg_pos={0 0}, upload=462809]: Uploading piece... Dec 27 15:08:07.296653387 (527h39m7.167736432s) [upload=462809]: Next returned nil segment Dec 27 15:08:07.296678645 (527h39m7.16776169s) [upload=462809]: Waiting for error group managing segments... Dec 27 15:08:07.584486294 (527h39m7.455569329s) [noise=true , address=145.79.227.29:4943 , piece_id=HMOJ4YN6V37Y73FJZ53M5SSRO3ADTQPVVFF7Y7RVCZ3SIODCV3BA, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.584840308 (527h39m7.455923343s) [noise=true , address=145.79.227.45:4230 , piece_id=BBZ4NU6EFFH2G56TPZUCDYX4AEECF2O24MUPOHAXTC7L5ZJSNKYA, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.590000534 (527h39m7.461083579s) [noise=true , address=145.79.227.3:4151 , piece_id=CMTD3ORK3LXXIZUNOWUUYQHLW3I6FX3MTYXVNUHUDYYK2ZCJTE6Q, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.591668069 (527h39m7.462751115s) [noise=true , address=145.79.227.7:4294 , piece_id=NI5ICWVHL2VNC3UTJRMVRC3BUGOFDQ6W3AXURW2X53G4S3C426YQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.592081948 (527h39m7.463164993s) [noise=true , address=145.79.227.52:4445 , piece_id=DZN75CIDAUOWAAERSBJV3SMTBGDJBWFRJRIVL3NSJXQOSBPNPQTQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.592530171 (527h39m7.463613216s) [noise=true , address=145.79.227.37:4565 , piece_id=3N3XYM576355JXLEJI75QH4IQ57RMEDE4FVTEGLAJD3QMKKHQ3HQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.601578741 (527h39m7.472661776s) [noise=true , address=145.79.227.20:4438 , piece_id=4Q5AD4KN5HFTAXNMMKSF6N7X66YWLSWFNEFZHPQQ5YP2IFHF7PIQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.612989379 (527h39m7.484072424s) [noise=true , address=145.79.227.23:4546 , piece_id=3HECX7NWRC2YDT4AOOGWBKD4WGJSCXJ2ETQIPRVCSOCJBFVOEZ6A, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.614224171 (527h39m7.485307216s) [noise=true , address=145.79.227.8:4329 , piece_id=GMOGZCIZZ32562BOISN6JI2QQKXCRDDYHSGLTT6KFELSIMQAOF2Q, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.814415883 (527h39m7.685498918s) [noise=true , address=77.90.31.14:4661 , piece_id=XBXENLARRGNHVUUTD5ILUE6LTIZDE77BCFNHXHPEZTY4CGH7254Q, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.865179466 (527h39m7.736262501s) [noise=true , address=45.91.116.14:4192 , piece_id=XGK2R4EJGAVV7XH7TY43W4XM3KSM7VMNJROAZLWPZB6Z2MQL4Y6Q, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.932797994 (527h39m7.803881039s) [noise=true , address=45.91.116.10:4083 , piece_id=4WZOXQACUODLZ4PU65TVUMTEMDJ6VABH7UTZBY555IQOFPX5NKMQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.934328599 (527h39m7.805411644s) [noise=true , address=45.91.116.6:4975 , piece_id=JSX4WITHXMMUFT7UNI6M3B6UD6LLUUQWNICAHZFDY6AOGAYLWRKQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: <nil> Dec 27 15:08:07.934343448 (527h39m7.805426503s) [seg_pos={0 0}, upload=462809]: Segment reached optimal threshold of 13 pieces. Dec 27 15:08:07.934576811 (527h39m7.805659856s) [noise=true , address=45.91.116.4:4937 , piece_id=4LHNTITWH4GVYH4FCR4377UMVXG4S6B37E3W7DQJJEYWWYMAPSTQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:56734->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:56734->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56734->45.91.116.4:4937: use of closed network connection Dec 27 15:08:07.934585679 (527h39m7.805668714s) [noise=true , address=45.91.116.4:4937 , piece_id=4LHNTITWH4GVYH4FCR4377UMVXG4S6B37E3W7DQJJEYWWYMAPSTQ, seg_pos={0 0}, upload=462809]: Long tail cancellation due to optimal threshold Dec 27 15:08:07.934596088 (527h39m7.805679123s) [noise=true , address=45.91.116.11:4120 , piece_id=HV23MQWNDNI2QEBVR6JPPMP22PRGIBOEGGWFLUSQT5SPAPPRTANQ, seg_pos={0 0}, upload=462809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:39692->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:39692->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39692->45.91.116.11:4120: use of closed network connection Dec 27 15:08:07.934601438 (527h39m7.805684473s) [noise=true , address=45.91.116.11:4120 , piece_id=HV23MQWNDNI2QEBVR6JPPMP22PRGIBOEGGWFLUSQT5SPAPPRTANQ, seg_pos={0 0}, upload=462809]: Long tail cancellation due to optimal threshold Dec 27 15:08:07.934906750 (527h39m7.805989815s) [noise=true , address=77.90.31.19:4841 , piece_id=QM3DYB5BHN3PEWRD4WRWGP77XPMHI54DIKYBQAXTFJIRL3VUDR6A, seg_pos={0 0}, upload=462809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:43360->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 27 15:08:07.934916508 (527h39m7.805999543s) [noise=true , address=77.90.31.19:4841 , piece_id=QM3DYB5BHN3PEWRD4WRWGP77XPMHI54DIKYBQAXTFJIRL3VUDR6A, seg_pos={0 0}, upload=462809]: Long tail cancellation due to optimal threshold Dec 27 15:08:07.934936046 (527h39m7.806019091s) [seg_pos={0 0}, upload=462809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:08:07.935022640 (527h39m7.806105685s) [upload=462809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:08:07.935028992 (527h39m7.806112037s) [upload=462809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:08:08.236777799 (527h39m8.107860844s) [upload=462809]: Done uploading segments... Dec 27 15:08:08.236806484 (527h39m8.107889529s) [upload=462809]: Upload finished. err: <nil> Duration: 30.002978272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X..X...........X..............X....X...........] duration: 20.605237148s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:08:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 20.605296442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X.........X.X....X...X.....X..XX.......X..X.X..X...XXXX] duration: 30.005273824s trace: https://jaeger.dev.storj.io/trace/74a8b547f5939c5e FAIL (2025-12-27 15:08:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: R3V53U3GT3JKBXWSBF2XANHCQFNEKVKZL4OSXILGW7NFB5CVQPZQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 36SCY45XBGHBF6ESJUGMFM3D5DBDOPH5VSKMTPXZZGB76EELKLTQ) context deadline exceeded ======LOGS====== Dec 27 15:07:35.096821598 (527h38m34.967904643s) [upload=462802]: Starting upload Dec 27 15:07:35.096933220 (527h38m34.968016255s) [upload=462802]: Done starting upload Dec 27 15:07:35.097043310 (527h38m34.968126356s) [upload=462802]: Uploading segments... Dec 27 15:07:35.097136577 (527h38m34.968219612s) [upload=462802]: Got next segment. Inline: false Dec 27 15:07:35.097169941 (527h38m34.968252976s) [upload=462802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:07:35.097175261 (527h38m34.968258296s) [upload=462802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:07:35.342724402 (527h38m35.213807447s) [seg_pos={0 0}, upload=462802]: Begin upload segment... Dec 27 15:07:35.342772433 (527h38m35.213855468s) [seg_pos={0 0}, upload=462802]: Done begin upload segment. Dec 27 15:07:35.342813873 (527h38m35.213896908s) [noise=true , address=45.91.116.6:4975 , piece_id=IEVBVISQ3X2XEJAKWAYYTBC7JOY7UGW6JPIWAGBEMRNAV5G3FU6Q, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.342893965 (527h38m35.213977s) [noise=true , address=145.79.227.18:4007 , piece_id=ALUO2GIM3PLSX4FDUIIKFUNSMWZZZAS5Z7HUUDG4ZOEXEATWWJHQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.343452609 (527h38m35.214535654s) [noise=true , address=77.90.31.17:4770 , piece_id=FGBKNVEMQRM5PBBFOCL65A5BKMYFMDV4D56DW6WENC3JP2ABNBAQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.343824257 (527h38m35.214907302s) [noise=true , address=145.79.227.48:4302 , piece_id=YUASRH2KMF2WVXFFRA36OOGGOJUA4YZU46YYKWQWES6OPWAZWVBQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.344172951 (527h38m35.215255986s) [noise=true , address=145.79.227.42:4123 , piece_id=ZAYLJD4CKZOKBM7UEQRIDBNWB4IFWCIUFJLR57DDTZICN34PUTPQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.344482510 (527h38m35.215565545s) [noise=true , address=145.79.227.7:4295 , piece_id=Y5MOFCLYOIMAQC3AA4HU6XAYXHXLH42XGLMJZBWTVEMDI4KOYODA, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.344782872 (527h38m35.215865907s) [noise=true , address=45.91.116.5:4298 , piece_id=R3V53U3GT3JKBXWSBF2XANHCQFNEKVKZL4OSXILGW7NFB5CVQPZQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.345095036 (527h38m35.216178081s) [noise=true , address=77.90.31.11:4555 , piece_id=Z64JKKRQXYFUY6RWUFCIRVBYP4ISNDUQ6SBRP7NAPFBBCLDY5XAQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.345441937 (527h38m35.216524972s) [noise=true , address=145.79.227.21:4475 , piece_id=PI6AFCSWS5MMVAI4VWZOPUW5K6ZS5J5DHWJHSXHUWNAEF6BSDEBA, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.345849784 (527h38m35.216932819s) [noise=true , address=145.79.227.12:4767 , piece_id=XJVBE747EAEHMRUQVU34VGNSWHBLCNHOI4SXJQBDEQOXV3NDYJXA, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.346410631 (527h38m35.217493666s) [noise=true , address=145.79.227.46:4603 , piece_id=43D4P4I52CAEVF5PW73RGRHD7QPIOHCHCECPLIKOE22KCQX6Q2PQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.346793771 (527h38m35.217876806s) [noise=true , address=45.91.116.7:4009 , piece_id=DBHJSDIIVNFGCRBCGHHWOTBEYFXUQJJYCOUUOIR4GFM3TDTIWPPQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.347087600 (527h38m35.218170635s) [noise=true , address=45.91.116.14:4189 , piece_id=36SCY45XBGHBF6ESJUGMFM3D5DBDOPH5VSKMTPXZZGB76EELKLTQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.347458156 (527h38m35.218541191s) [noise=true , address=77.90.31.10:4520 , piece_id=F55VD4PZKLQDKMIJZEEZQKXLISRPBEMZ7BMNAGV25XHM4X5O36ZQ, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.347936637 (527h38m35.219019672s) [noise=true , address=145.79.227.32:4834 , piece_id=BDXF327H4VFMLM64REY2EXWFXJ275MRMY4JT3O3ADYSU2KVQQU7Q, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:35.348345626 (527h38m35.219428661s) [noise=true , address=145.79.227.35:4483 , piece_id=ZSKSFG4R5ZCFHAZ6736SWXADDKQQPXG6CRBM2OEIPNH455LSI72Q, seg_pos={0 0}, upload=462802]: Uploading piece... Dec 27 15:07:37.054679222 (527h38m36.925762257s) [upload=462802]: Next returned nil segment Dec 27 15:07:37.054695091 (527h38m36.925778126s) [upload=462802]: Waiting for error group managing segments... Dec 27 15:07:37.348034905 (527h38m37.21911795s) [noise=true , address=145.79.227.7:4295 , piece_id=Y5MOFCLYOIMAQC3AA4HU6XAYXHXLH42XGLMJZBWTVEMDI4KOYODA, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.351236392 (527h38m37.222319447s) [noise=true , address=145.79.227.12:4767 , piece_id=XJVBE747EAEHMRUQVU34VGNSWHBLCNHOI4SXJQBDEQOXV3NDYJXA, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.352754472 (527h38m37.223837507s) [noise=true , address=145.79.227.32:4834 , piece_id=BDXF327H4VFMLM64REY2EXWFXJ275MRMY4JT3O3ADYSU2KVQQU7Q, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.353790244 (527h38m37.224873279s) [noise=true , address=145.79.227.42:4123 , piece_id=ZAYLJD4CKZOKBM7UEQRIDBNWB4IFWCIUFJLR57DDTZICN34PUTPQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.361644911 (527h38m37.232727956s) [noise=true , address=145.79.227.46:4603 , piece_id=43D4P4I52CAEVF5PW73RGRHD7QPIOHCHCECPLIKOE22KCQX6Q2PQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.362101671 (527h38m37.233184706s) [noise=true , address=145.79.227.18:4007 , piece_id=ALUO2GIM3PLSX4FDUIIKFUNSMWZZZAS5Z7HUUDG4ZOEXEATWWJHQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.367070923 (527h38m37.238153968s) [noise=true , address=145.79.227.21:4475 , piece_id=PI6AFCSWS5MMVAI4VWZOPUW5K6ZS5J5DHWJHSXHUWNAEF6BSDEBA, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.367169420 (527h38m37.238252455s) [noise=true , address=145.79.227.35:4483 , piece_id=ZSKSFG4R5ZCFHAZ6736SWXADDKQQPXG6CRBM2OEIPNH455LSI72Q, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.370739529 (527h38m37.241822564s) [noise=true , address=145.79.227.48:4302 , piece_id=YUASRH2KMF2WVXFFRA36OOGGOJUA4YZU46YYKWQWES6OPWAZWVBQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.661303707 (527h38m37.532386752s) [noise=true , address=77.90.31.10:4520 , piece_id=F55VD4PZKLQDKMIJZEEZQKXLISRPBEMZ7BMNAGV25XHM4X5O36ZQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.732151274 (527h38m37.603234319s) [noise=true , address=45.91.116.14:4189 , piece_id=36SCY45XBGHBF6ESJUGMFM3D5DBDOPH5VSKMTPXZZGB76EELKLTQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:37.953783934 (527h38m37.824866979s) [noise=true , address=45.91.116.5:4298 , piece_id=R3V53U3GT3JKBXWSBF2XANHCQFNEKVKZL4OSXILGW7NFB5CVQPZQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:38.288899726 (527h38m38.159982771s) [noise=true , address=77.90.31.17:4770 , piece_id=FGBKNVEMQRM5PBBFOCL65A5BKMYFMDV4D56DW6WENC3JP2ABNBAQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: <nil> Dec 27 15:07:38.288925426 (527h38m38.160008461s) [seg_pos={0 0}, upload=462802]: Segment reached optimal threshold of 13 pieces. Dec 27 15:07:38.289158038 (527h38m38.160241103s) [noise=true , address=45.91.116.6:4975 , piece_id=IEVBVISQ3X2XEJAKWAYYTBC7JOY7UGW6JPIWAGBEMRNAV5G3FU6Q, seg_pos={0 0}, upload=462802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 15:07:38.289167877 (527h38m38.160250922s) [noise=true , address=45.91.116.6:4975 , piece_id=IEVBVISQ3X2XEJAKWAYYTBC7JOY7UGW6JPIWAGBEMRNAV5G3FU6Q, seg_pos={0 0}, upload=462802]: Long tail cancellation due to optimal threshold Dec 27 15:07:38.289293746 (527h38m38.160376791s) [noise=true , address=77.90.31.11:4555 , piece_id=Z64JKKRQXYFUY6RWUFCIRVBYP4ISNDUQ6SBRP7NAPFBBCLDY5XAQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:55652->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:07:38.289304608 (527h38m38.160387673s) [noise=true , address=77.90.31.11:4555 , piece_id=Z64JKKRQXYFUY6RWUFCIRVBYP4ISNDUQ6SBRP7NAPFBBCLDY5XAQ, seg_pos={0 0}, upload=462802]: Long tail cancellation due to optimal threshold Dec 27 15:07:38.289337860 (527h38m38.160420905s) [noise=true , address=45.91.116.7:4009 , piece_id=DBHJSDIIVNFGCRBCGHHWOTBEYFXUQJJYCOUUOIR4GFM3TDTIWPPQ, seg_pos={0 0}, upload=462802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 15:07:38.289345736 (527h38m38.160428781s) [noise=true , address=45.91.116.7:4009 , piece_id=DBHJSDIIVNFGCRBCGHHWOTBEYFXUQJJYCOUUOIR4GFM3TDTIWPPQ, seg_pos={0 0}, upload=462802]: Long tail cancellation due to optimal threshold Dec 27 15:07:38.289368038 (527h38m38.160451083s) [seg_pos={0 0}, upload=462802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:07:38.289460724 (527h38m38.16054378s) [upload=462802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:07:38.289468509 (527h38m38.160551554s) [upload=462802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:07:38.586102525 (527h38m38.45718556s) [upload=462802]: Done uploading segments... Dec 27 15:07:38.586131671 (527h38m38.457214736s) [upload=462802]: Upload finished. err: <nil> Duration: 30.005297578s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 13.032221174s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 13.032252494s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XX] duration: 5.001552745s trace: https://jaeger.dev.storj.io/trace/21e8a60eb58c2a90 FAIL (2025-12-27 15:07:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001567041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.............X..X...........X..............X....X..........] duration: 25.788397529s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:07:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 25.788414632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [X.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.X] duration: 5.001289794s trace: https://jaeger.dev.storj.io/trace/3679206f328477db FAIL (2025-12-27 15:07:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001315953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X....X.........X.X....X...X.....X..XX.......X..X.X..X...XXX] duration: 30.003946658s trace: https://jaeger.dev.storj.io/trace/6b5f87fff66a17ac FAIL (2025-12-27 15:07:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MBIH4EM66MS2O4RAVCDERO7TEI5JEGZYPZG5CPRADBGZBEH2XKSQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: W75SMYIXIAQWVCGUHUWXV6MIDK4BFBS56A6HRKIEE5VQ3GB7S7CQ) context deadline exceeded ======LOGS====== gments... Dec 27 15:06:36.775262165 (527h37m36.64634521s) [upload=462788]: Got next segment. Inline: false Dec 27 15:06:36.775335496 (527h37m36.646418541s) [upload=462788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:06:36.775343972 (527h37m36.646427007s) [upload=462788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:06:37.019442561 (527h37m36.890525606s) [seg_pos={0 0}, upload=462788]: Begin upload segment... Dec 27 15:06:37.019501864 (527h37m36.890584909s) [seg_pos={0 0}, upload=462788]: Done begin upload segment. Dec 27 15:06:37.019540096 (527h37m36.890623142s) [noise=true , address=145.79.227.42:4121 , piece_id=5TAOSQSBJDI4C3ZKXBGQ2IR2RDKQXGHV72DVYPU3IYN7ENWWAFKA, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.020034147 (527h37m36.891117192s) [noise=true , address=145.79.227.48:4301 , piece_id=6AGQQ7RT64ICATLMOSKCBWYWJQAE7PDREPD4ZZ6CTPZRA22U2FEA, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.020197558 (527h37m36.891280594s) [noise=true , address=145.79.227.3:4150 , piece_id=OIRLHTGHLNL6D4JGRZ6PP2HWLR5CN5HNVYVV7ZCMVOV37USJQ36Q, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.020870990 (527h37m36.891954036s) [noise=true , address=45.91.116.6:4974 , piece_id=QCU5HPRTTHRUZUJ6OCUZ35UIWHCZ3T6XET5MUX2HTKR6X3TFZOIQ, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.020954049 (527h37m36.892037084s) [noise=true , address=145.79.227.20:4438 , piece_id=WE657R6BWR5DTAVTK4TAZHNPOEDPID3JG4UT3YJLJLTKHXH6BI5Q, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.021292733 (527h37m36.892375778s) [noise=true , address=45.91.116.13:4156 , piece_id=MKEF4EWJIFOQ3QNAY4D77P5L44ZIOWCSQU7C52WPLSCXG4WLAN4A, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.021466174 (527h37m36.892549209s) [noise=true , address=145.79.227.5:4221 , piece_id=BCUUY2JL6CKZ3I2QPYQNOJYIZ2QKJPCWGR2Z2RZ4X6C34EDR65PQ, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.021782546 (527h37m36.892865581s) [noise=true , address=145.79.227.45:4230 , piece_id=36DTR554CULOI3SPXNHSQQYMWNU5J3QQZRDJK655HAMY6CCRYHQA, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.021907725 (527h37m36.89299076s) [noise=true , address=145.79.227.32:4835 , piece_id=K7JITDHCF5BOCE67ZQA766ACFKPZR6KHREBWZXVSUJCKIDKETPOQ, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.022330410 (527h37m36.893413445s) [noise=true , address=77.90.31.14:4664 , piece_id=W75SMYIXIAQWVCGUHUWXV6MIDK4BFBS56A6HRKIEE5VQ3GB7S7CQ, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.022348014 (527h37m36.893431059s) [noise=true , address=145.79.227.13:4802 , piece_id=ONWSRKOWR5VVMGOEPXOF4EFA54NR6SPZKCBWWNNC5GGC3NS6DEZA, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.022673182 (527h37m36.893756217s) [noise=true , address=145.79.227.8:4329 , piece_id=NJ5B3NCMY2SP6RZPIYZKNOXKQXSNX5KKOGQIJSB5GJLQFQH7CXKQ, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.022835762 (527h37m36.893918807s) [noise=true , address=77.90.31.18:4808 , piece_id=YSVEGSEOGOULUMDT3Q4GSTJF42D5KKIBB6WALHZKWN5YBTPMVSZQ, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.022997319 (527h37m36.894080354s) [noise=true , address=45.91.116.11:4117 , piece_id=MBIH4EM66MS2O4RAVCDERO7TEI5JEGZYPZG5CPRADBGZBEH2XKSQ, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.023277893 (527h37m36.894360929s) [noise=true , address=145.79.227.29:4941 , piece_id=VBA3M24LL4HB4LTV4FOWM2F2H64YQVKMRZFV2DJQWZOOM5OO2LGA, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:37.023322730 (527h37m36.894405765s) [noise=true , address=45.91.116.12:4225 , piece_id=TZPOF4WYWTVWEBKBP3U5E3M2QLD3TUVBC3MOXWP2IWTYIWR4TASA, seg_pos={0 0}, upload=462788]: Uploading piece... Dec 27 15:06:38.741011747 (527h37m38.612094782s) [upload=462788]: Next returned nil segment Dec 27 15:06:38.741029201 (527h37m38.612112276s) [upload=462788]: Waiting for error group managing segments... Dec 27 15:06:39.025025753 (527h37m38.896108788s) [noise=true , address=145.79.227.3:4150 , piece_id=OIRLHTGHLNL6D4JGRZ6PP2HWLR5CN5HNVYVV7ZCMVOV37USJQ36Q, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.027798592 (527h37m38.898881628s) [noise=true , address=145.79.227.13:4802 , piece_id=ONWSRKOWR5VVMGOEPXOF4EFA54NR6SPZKCBWWNNC5GGC3NS6DEZA, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.028241516 (527h37m38.899324561s) [noise=true , address=145.79.227.45:4230 , piece_id=36DTR554CULOI3SPXNHSQQYMWNU5J3QQZRDJK655HAMY6CCRYHQA, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.030547337 (527h37m38.901630382s) [noise=true , address=145.79.227.29:4941 , piece_id=VBA3M24LL4HB4LTV4FOWM2F2H64YQVKMRZFV2DJQWZOOM5OO2LGA, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.030583215 (527h37m38.90166625s) [noise=true , address=145.79.227.20:4438 , piece_id=WE657R6BWR5DTAVTK4TAZHNPOEDPID3JG4UT3YJLJLTKHXH6BI5Q, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.031054222 (527h37m38.902137267s) [noise=true , address=145.79.227.42:4121 , piece_id=5TAOSQSBJDI4C3ZKXBGQ2IR2RDKQXGHV72DVYPU3IYN7ENWWAFKA, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.042557306 (527h37m38.913640351s) [noise=true , address=145.79.227.48:4301 , piece_id=6AGQQ7RT64ICATLMOSKCBWYWJQAE7PDREPD4ZZ6CTPZRA22U2FEA, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.046809714 (527h37m38.917892759s) [noise=true , address=145.79.227.32:4835 , piece_id=K7JITDHCF5BOCE67ZQA766ACFKPZR6KHREBWZXVSUJCKIDKETPOQ, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.050504349 (527h37m38.921587384s) [noise=true , address=145.79.227.5:4221 , piece_id=BCUUY2JL6CKZ3I2QPYQNOJYIZ2QKJPCWGR2Z2RZ4X6C34EDR65PQ, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.055452180 (527h37m38.926535225s) [noise=true , address=145.79.227.8:4329 , piece_id=NJ5B3NCMY2SP6RZPIYZKNOXKQXSNX5KKOGQIJSB5GJLQFQH7CXKQ, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.318402697 (527h37m39.189485742s) [noise=true , address=77.90.31.14:4664 , piece_id=W75SMYIXIAQWVCGUHUWXV6MIDK4BFBS56A6HRKIEE5VQ3GB7S7CQ, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.352388552 (527h37m39.223471587s) [noise=true , address=45.91.116.11:4117 , piece_id=MBIH4EM66MS2O4RAVCDERO7TEI5JEGZYPZG5CPRADBGZBEH2XKSQ, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.376578625 (527h37m39.24766166s) [noise=true , address=45.91.116.13:4156 , piece_id=MKEF4EWJIFOQ3QNAY4D77P5L44ZIOWCSQU7C52WPLSCXG4WLAN4A, seg_pos={0 0}, upload=462788]: Done uploading piece. err: <nil> Dec 27 15:06:39.376596379 (527h37m39.247679414s) [seg_pos={0 0}, upload=462788]: Segment reached optimal threshold of 13 pieces. Dec 27 15:06:39.376776341 (527h37m39.247859376s) [noise=true , address=45.91.116.12:4225 , piece_id=TZPOF4WYWTVWEBKBP3U5E3M2QLD3TUVBC3MOXWP2IWTYIWR4TASA, seg_pos={0 0}, upload=462788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:50702->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:50702->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50702->45.91.116.12:4225: use of closed network connection Dec 27 15:06:39.376782613 (527h37m39.247865648s) [noise=true , address=45.91.116.12:4225 , piece_id=TZPOF4WYWTVWEBKBP3U5E3M2QLD3TUVBC3MOXWP2IWTYIWR4TASA, seg_pos={0 0}, upload=462788]: Long tail cancellation due to optimal threshold Dec 27 15:06:39.376982163 (527h37m39.248065228s) [noise=true , address=77.90.31.18:4808 , piece_id=YSVEGSEOGOULUMDT3Q4GSTJF42D5KKIBB6WALHZKWN5YBTPMVSZQ, seg_pos={0 0}, upload=462788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:48482->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:48482->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48482->77.90.31.18:4808: use of closed network connection Dec 27 15:06:39.376993254 (527h37m39.248076299s) [noise=true , address=77.90.31.18:4808 , piece_id=YSVEGSEOGOULUMDT3Q4GSTJF42D5KKIBB6WALHZKWN5YBTPMVSZQ, seg_pos={0 0}, upload=462788]: Long tail cancellation due to optimal threshold Dec 27 15:06:39.376998304 (527h37m39.248081349s) [noise=true , address=45.91.116.6:4974 , piece_id=QCU5HPRTTHRUZUJ6OCUZ35UIWHCZ3T6XET5MUX2HTKR6X3TFZOIQ, seg_pos={0 0}, upload=462788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 15:06:39.377018112 (527h37m39.248101147s) [noise=true , address=45.91.116.6:4974 , piece_id=QCU5HPRTTHRUZUJ6OCUZ35UIWHCZ3T6XET5MUX2HTKR6X3TFZOIQ, seg_pos={0 0}, upload=462788]: Long tail cancellation due to optimal threshold Dec 27 15:06:39.377163749 (527h37m39.248246784s) [seg_pos={0 0}, upload=462788]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:06:39.377264180 (527h37m39.248347215s) [upload=462788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:06:39.377269751 (527h37m39.248352786s) [upload=462788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:06:39.690166975 (527h37m39.56125003s) [upload=462788]: Done uploading segments... Dec 27 15:06:39.690191832 (527h37m39.561274877s) [upload=462788]: Upload finished. err: <nil> Duration: 30.003959412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.............X..X...........X..............X....X.........] duration: 45.722487789s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 45.722600003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 9.456315798s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:06:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 9.45634315s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....X.........X.X....X...X.....X..XX.......X..X.X..X...XX] duration: 30.005120389s trace: https://jaeger.dev.storj.io/trace/6497bd92fd125e9d FAIL (2025-12-27 15:06:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 15:06:06.530207418 (527h37m6.401290463s) [upload=462780]: Starting upload Dec 27 15:06:06.530297489 (527h37m6.401380534s) [upload=462780]: Done starting upload Dec 27 15:06:06.530495226 (527h37m6.401578271s) [upload=462780]: Uploading segments... Dec 27 15:06:06.530630955 (527h37m6.401714s) [upload=462780]: Got next segment. Inline: false Dec 27 15:06:06.530674227 (527h37m6.401757262s) [upload=462780]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:06:06.530680508 (527h37m6.401763544s) [upload=462780]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:06:06.775815641 (527h37m6.646898696s) [seg_pos={0 0}, upload=462780]: Begin upload segment... Dec 27 15:06:06.775880515 (527h37m6.64696356s) [seg_pos={0 0}, upload=462780]: Done begin upload segment. Dec 27 15:06:06.775953414 (527h37m6.647036459s) [noise=true , address=77.90.31.10:4518 , piece_id=MEP2B3VSDAQT3GOISNNVB6NXGH7EK7JDQHFXI7TZ26ZPLCR7MWZA, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.776015412 (527h37m6.647098457s) [noise=true , address=145.79.227.11:4730 , piece_id=YZAKMS3R2WVHWNBKWW5VSR436VR5MJEXMMM563YNNQ7LJFUG5NAQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.776597980 (527h37m6.647681025s) [noise=true , address=145.79.227.40:4051 , piece_id=C4PAW4IXQ3VJX47CPSDOMQN2LLEF33TRR2V2GFUARD3BP6AJE4JA, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.776792972 (527h37m6.647876017s) [noise=true , address=45.91.116.12:4227 , piece_id=VUI7MSJDYJXDYDHEGTQP3FMX3KFKKXKSCTMG375T2DE3B2K2TYSQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.777089187 (527h37m6.648172232s) [noise=true , address=145.79.227.21:4474 , piece_id=GQ7L2H37GUGPQHKUH43IRSF6GXP22JOKNWBTLDAQE6VAXEMZT53A, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.777290639 (527h37m6.648373684s) [noise=true , address=45.91.116.6:4973 , piece_id=D2TNNMKCSNHMULLXG44RVDQOHC2N52TIOMZK4GNB6QDNQNMYXZ6A, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.777540767 (527h37m6.648623812s) [noise=true , address=145.79.227.10:4403 , piece_id=RBHGUGQUK4JOGUOI7AKQFQYPD2YH5BA4GU3AZKH22UDWYYVZ4KDA, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.777780622 (527h37m6.648863657s) [noise=true , address=145.79.227.42:4123 , piece_id=MS425GSZPZ6AJSVJMD7AMLWAL5S6IAP47JPP3FID4WQEGBPUVRDQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.778088729 (527h37m6.649171774s) [noise=true , address=45.91.116.14:4189 , piece_id=D3NRSZOALDMKDCLHRCTBHWHAFN22ESBOE2FNDTIZN2JN4RUENNIA, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.778513549 (527h37m6.649596594s) [noise=true , address=145.79.227.29:4943 , piece_id=YKA5I3SFYHKPJ5ZEYITMTUNXAVYY3D3TFGO2NZCQ3JLC2ZS5LI7A, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.778918209 (527h37m6.650001244s) [noise=true , address=145.79.227.16:4911 , piece_id=5QUTNLNAOMZ4KXUU76TFUVNZYOMIDENM7TESP7OC2BSGBP2432CQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.779349070 (527h37m6.650432115s) [noise=true , address=77.90.31.13:4629 , piece_id=PSHW3JVMSMWKMW524FDTQLWJUOUMRVTSA3Y26SFDXJ2JHZVSIOQQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.779855944 (527h37m6.65093899s) [noise=true , address=145.79.227.5:4221 , piece_id=3QF4D55L7GCR5JRZQ744CX5YPT7N7I4BS5VYXMGTD4ELMFVIINWQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.780232993 (527h37m6.651316028s) [noise=true , address=145.79.227.12:4765 , piece_id=TUB4OOOCXLNIUWVOV3NZLUGG4MMTRQ7J4C4S4UGA7SRTIHFDGTLQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.780797788 (527h37m6.651880833s) [noise=true , address=145.79.227.50:4373 , piece_id=GAL5IAZDMWVD6366WAYWCOBAGKDBKALWEPD3HOKKV634664WJB5Q, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:06.781200966 (527h37m6.652284011s) [noise=true , address=145.79.227.17:4971 , piece_id=MWLSPGQ34AL767LHKKFTHPDOESBEIPOC7NBTUCRKY5XF4B5GVDAQ, seg_pos={0 0}, upload=462780]: Uploading piece... Dec 27 15:06:08.391150188 (527h37m8.262233223s) [upload=462780]: Next returned nil segment Dec 27 15:06:08.391170056 (527h37m8.262253101s) [upload=462780]: Waiting for error group managing segments... Dec 27 15:06:08.779769521 (527h37m8.650852566s) [noise=true , address=145.79.227.16:4911 , piece_id=5QUTNLNAOMZ4KXUU76TFUVNZYOMIDENM7TESP7OC2BSGBP2432CQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.786709607 (527h37m8.657792652s) [noise=true , address=145.79.227.17:4971 , piece_id=MWLSPGQ34AL767LHKKFTHPDOESBEIPOC7NBTUCRKY5XF4B5GVDAQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.786924947 (527h37m8.658007992s) [noise=true , address=145.79.227.11:4730 , piece_id=YZAKMS3R2WVHWNBKWW5VSR436VR5MJEXMMM563YNNQ7LJFUG5NAQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.787929931 (527h37m8.659012976s) [noise=true , address=145.79.227.21:4474 , piece_id=GQ7L2H37GUGPQHKUH43IRSF6GXP22JOKNWBTLDAQE6VAXEMZT53A, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.790941995 (527h37m8.662025031s) [noise=true , address=145.79.227.42:4123 , piece_id=MS425GSZPZ6AJSVJMD7AMLWAL5S6IAP47JPP3FID4WQEGBPUVRDQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.791999730 (527h37m8.663082775s) [noise=true , address=145.79.227.29:4943 , piece_id=YKA5I3SFYHKPJ5ZEYITMTUNXAVYY3D3TFGO2NZCQ3JLC2ZS5LI7A, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.803024974 (527h37m8.674108009s) [noise=true , address=145.79.227.5:4221 , piece_id=3QF4D55L7GCR5JRZQ744CX5YPT7N7I4BS5VYXMGTD4ELMFVIINWQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.803151055 (527h37m8.67423409s) [noise=true , address=145.79.227.10:4403 , piece_id=RBHGUGQUK4JOGUOI7AKQFQYPD2YH5BA4GU3AZKH22UDWYYVZ4KDA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.805604276 (527h37m8.676687311s) [noise=true , address=145.79.227.50:4373 , piece_id=GAL5IAZDMWVD6366WAYWCOBAGKDBKALWEPD3HOKKV634664WJB5Q, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:08.809331353 (527h37m8.680414398s) [noise=true , address=145.79.227.12:4765 , piece_id=TUB4OOOCXLNIUWVOV3NZLUGG4MMTRQ7J4C4S4UGA7SRTIHFDGTLQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:09.041305505 (527h37m8.91238854s) [noise=true , address=77.90.31.10:4518 , piece_id=MEP2B3VSDAQT3GOISNNVB6NXGH7EK7JDQHFXI7TZ26ZPLCR7MWZA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:09.041878486 (527h37m8.912961531s) [noise=true , address=45.91.116.6:4973 , piece_id=D2TNNMKCSNHMULLXG44RVDQOHC2N52TIOMZK4GNB6QDNQNMYXZ6A, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:09.572028099 (527h37m9.443111144s) [noise=true , address=45.91.116.14:4189 , piece_id=D3NRSZOALDMKDCLHRCTBHWHAFN22ESBOE2FNDTIZN2JN4RUENNIA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil> Dec 27 15:06:09.572048749 (527h37m9.443131794s) [seg_pos={0 0}, upload=462780]: Segment reached optimal threshold of 13 pieces. Dec 27 15:06:09.572261314 (527h37m9.443344349s) [noise=true , address=77.90.31.13:4629 , piece_id=PSHW3JVMSMWKMW524FDTQLWJUOUMRVTSA3Y26SFDXJ2JHZVSIOQQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 27 15:06:09.572267235 (527h37m9.44335028s) [noise=true , address=77.90.31.13:4629 , piece_id=PSHW3JVMSMWKMW524FDTQLWJUOUMRVTSA3Y26SFDXJ2JHZVSIOQQ, seg_pos={0 0}, upload=462780]: Long tail cancellation due to optimal threshold Dec 27 15:06:09.572503635 (527h37m9.44358668s) [noise=true , address=45.91.116.12:4227 , piece_id=VUI7MSJDYJXDYDHEGTQP3FMX3KFKKXKSCTMG375T2DE3B2K2TYSQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:56564->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:56564->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56564->45.91.116.12:4227: use of closed network connection Dec 27 15:06:09.572513564 (527h37m9.443596609s) [noise=true , address=45.91.116.12:4227 , piece_id=VUI7MSJDYJXDYDHEGTQP3FMX3KFKKXKSCTMG375T2DE3B2K2TYSQ, seg_pos={0 0}, upload=462780]: Long tail cancellation due to optimal threshold Dec 27 15:06:09.572708013 (527h37m9.443791078s) [noise=true , address=145.79.227.40:4051 , piece_id=C4PAW4IXQ3VJX47CPSDOMQN2LLEF33TRR2V2GFUARD3BP6AJE4JA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 15:06:09.572747479 (527h37m9.443830544s) [noise=true , address=145.79.227.40:4051 , piece_id=C4PAW4IXQ3VJX47CPSDOMQN2LLEF33TRR2V2GFUARD3BP6AJE4JA, seg_pos={0 0}, upload=462780]: Long tail cancellation due to optimal threshold Dec 27 15:06:09.572779430 (527h37m9.443862475s) [seg_pos={0 0}, upload=462780]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:06:09.572914717 (527h37m9.443997762s) [upload=462780]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:06:09.572922412 (527h37m9.444005457s) [upload=462780]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:06:09.863899917 (527h37m9.734982962s) [upload=462780]: Done uploading segments... Dec 27 15:06:09.863923492 (527h37m9.735006537s) [upload=462780]: Upload finished. err: <nil> Duration: 30.00513657s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 10.770482278s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:06:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 10.770511633s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.............X..X...........X..............X....X........] duration: 29.892838803s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:06:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 29.892863679s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.] duration: 3.926139969s trace: https://jaeger.dev.storj.io/trace/677729cf7e56cee FAIL (2025-12-27 15:06:14 UTC) orbiter warning: context deadline exceeded Duration: 3.926168522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X....X.........X.X....X...X.....X..XX.......X..X.X..X...X] duration: 30.004069318s trace: https://jaeger.dev.storj.io/trace/2d2d647d95d7cd9c FAIL (2025-12-27 15:06:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SL7HZGRJRIKILJKXYP3KNJPSR5UV25IL26VQV5X44QAKHDNSKRAQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 6DX76LZSTIIAZNYYUOGOWJVGYIQBNXAS5QR6EOSFQSQDPE6N5JQA) context deadline exceeded ======LOGS====== 72]: Starting upload Dec 27 15:05:36.276416804 (527h36m36.147499839s) [upload=462772]: Done starting upload Dec 27 15:05:36.276486918 (527h36m36.147569953s) [upload=462772]: Uploading segments... Dec 27 15:05:36.276584193 (527h36m36.147667228s) [upload=462772]: Got next segment. Inline: false Dec 27 15:05:36.276628728 (527h36m36.147711763s) [upload=462772]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:05:36.276634228 (527h36m36.147717273s) [upload=462772]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:05:36.531893098 (527h36m36.402976153s) [seg_pos={0 0}, upload=462772]: Begin upload segment... Dec 27 15:05:36.531948665 (527h36m36.40303174s) [seg_pos={0 0}, upload=462772]: Done begin upload segment. Dec 27 15:05:36.532033796 (527h36m36.403116841s) [noise=true , address=145.79.227.39:4015 , piece_id=ZXMRGWDE4ZTFNMLS5BH2STTPBT3JMI4HULEQ3AUSEMHMDXTQOLDQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.532091036 (527h36m36.403174081s) [noise=true , address=145.79.227.48:4302 , piece_id=ASRGOO4LXNGVYNSJNT7XG7NUBAUNBV4SJT3RRGIA6M6JUO3MTH4A, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.532603941 (527h36m36.403686977s) [noise=true , address=45.91.116.8:4047 , piece_id=6DX76LZSTIIAZNYYUOGOWJVGYIQBNXAS5QR6EOSFQSQDPE6N5JQA, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.532651813 (527h36m36.403734848s) [noise=true , address=145.79.227.41:4085 , piece_id=4MHSTWNILDWAGBH4IEJX6SP5BQGV722R7S4XI2JFICINLG3HXTNQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.533010065 (527h36m36.4040931s) [noise=true , address=45.91.116.5:4297 , piece_id=KUKOTA75QZZ62NXEMZP4BSE6FCWBXEVKZL7ESJNNBDNWQMEZDVTQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.533632360 (527h36m36.404715405s) [noise=true , address=145.79.227.42:4122 , piece_id=C7KEWEPFJ2DVJMOACPOWTOAPPEOVFN2W5WF5LDRQX6O6JNN5BO3A, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.534261829 (527h36m36.405344864s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDGQKJEBT34LRQJQVS2LTFQGEUTM2BTE65R3ZSG26E33SCTCSVQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.534550869 (527h36m36.405633904s) [noise=true , address=145.79.227.10:4403 , piece_id=7N4MD6PL3KDB5VUF77QHSLSUUH4XSEG72RNKEGJKJGMTHYSZ7JWQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.535141293 (527h36m36.406224338s) [noise=true , address=145.79.227.21:4475 , piece_id=NEGHPLVNG75KAGVSCE6KAN735G4QWNKAAXUR3YP7PDLOFD2PYUGA, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.535403964 (527h36m36.406487009s) [noise=true , address=145.79.227.45:4229 , piece_id=23KSB6OFP5FGVQCRZXTT3YRZBTYVXFBWID2XAIOZJNWYQKYY24XA, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.535985501 (527h36m36.407068546s) [noise=true , address=77.90.31.21:4880 , piece_id=SL7HZGRJRIKILJKXYP3KNJPSR5UV25IL26VQV5X44QAKHDNSKRAQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.536266376 (527h36m36.407349411s) [noise=true , address=145.79.227.49:4339 , piece_id=F4K3FJO5H4V2PNRMSM5VRBG5JORGGV67Y2API6GAUGPRB2OWZOAA, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.536809870 (527h36m36.407892916s) [noise=true , address=145.79.227.47:4266 , piece_id=LP5M4RG56HTSEDHTHXS3TCN2KCEKCSI3MDZKD6RJ5SGSHHSMEK6Q, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.537171620 (527h36m36.408254665s) [noise=true , address=45.91.116.13:4156 , piece_id=WHB2LN2BM7ZRSTYZJIZ6GRIS3UGRSDD4SJUULO4LGVB7Z2OYWWTQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.537477792 (527h36m36.408560827s) [noise=true , address=45.91.116.6:4975 , piece_id=U2U4Z2WSMEJ426XIQDR4N5UAVRKCLU57C6FAD7ZP3CFM7CDQBXNA, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:36.537956093 (527h36m36.409039138s) [noise=true , address=145.79.227.37:4565 , piece_id=JTCAIUEG6H3GVZGJMHTX6Z3Y34L7WU3O7XJPWGF26AFUQIAMRDPQ, seg_pos={0 0}, upload=462772]: Uploading piece... Dec 27 15:05:38.158872068 (527h36m38.029955113s) [upload=462772]: Next returned nil segment Dec 27 15:05:38.158888891 (527h36m38.029971926s) [upload=462772]: Waiting for error group managing segments... Dec 27 15:05:38.543650418 (527h36m38.414733513s) [noise=true , address=145.79.227.39:4015 , piece_id=ZXMRGWDE4ZTFNMLS5BH2STTPBT3JMI4HULEQ3AUSEMHMDXTQOLDQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.546844460 (527h36m38.417927506s) [noise=true , address=145.79.227.45:4229 , piece_id=23KSB6OFP5FGVQCRZXTT3YRZBTYVXFBWID2XAIOZJNWYQKYY24XA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.548801847 (527h36m38.419884883s) [noise=true , address=145.79.227.48:4302 , piece_id=ASRGOO4LXNGVYNSJNT7XG7NUBAUNBV4SJT3RRGIA6M6JUO3MTH4A, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.549236926 (527h36m38.420319971s) [noise=true , address=145.79.227.37:4565 , piece_id=JTCAIUEG6H3GVZGJMHTX6Z3Y34L7WU3O7XJPWGF26AFUQIAMRDPQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.549324974 (527h36m38.420408019s) [noise=true , address=145.79.227.42:4122 , piece_id=C7KEWEPFJ2DVJMOACPOWTOAPPEOVFN2W5WF5LDRQX6O6JNN5BO3A, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.550006161 (527h36m38.421089196s) [noise=true , address=145.79.227.41:4085 , piece_id=4MHSTWNILDWAGBH4IEJX6SP5BQGV722R7S4XI2JFICINLG3HXTNQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.558655781 (527h36m38.429738816s) [noise=true , address=145.79.227.21:4475 , piece_id=NEGHPLVNG75KAGVSCE6KAN735G4QWNKAAXUR3YP7PDLOFD2PYUGA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.562459023 (527h36m38.433542068s) [noise=true , address=145.79.227.10:4403 , piece_id=7N4MD6PL3KDB5VUF77QHSLSUUH4XSEG72RNKEGJKJGMTHYSZ7JWQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.572453234 (527h36m38.443536279s) [noise=true , address=145.79.227.49:4339 , piece_id=F4K3FJO5H4V2PNRMSM5VRBG5JORGGV67Y2API6GAUGPRB2OWZOAA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.575805217 (527h36m38.446888262s) [noise=true , address=145.79.227.47:4266 , piece_id=LP5M4RG56HTSEDHTHXS3TCN2KCEKCSI3MDZKD6RJ5SGSHHSMEK6Q, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.815643705 (527h36m38.68672674s) [noise=true , address=77.90.31.21:4880 , piece_id=SL7HZGRJRIKILJKXYP3KNJPSR5UV25IL26VQV5X44QAKHDNSKRAQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:38.945441103 (527h36m38.816524148s) [noise=true , address=45.91.116.13:4156 , piece_id=WHB2LN2BM7ZRSTYZJIZ6GRIS3UGRSDD4SJUULO4LGVB7Z2OYWWTQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:39.142513235 (527h36m39.01359627s) [noise=true , address=45.91.116.8:4047 , piece_id=6DX76LZSTIIAZNYYUOGOWJVGYIQBNXAS5QR6EOSFQSQDPE6N5JQA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil> Dec 27 15:05:39.142539615 (527h36m39.013622661s) [seg_pos={0 0}, upload=462772]: Segment reached optimal threshold of 13 pieces. Dec 27 15:05:39.142828846 (527h36m39.013911901s) [noise=true , address=45.91.116.5:4297 , piece_id=KUKOTA75QZZ62NXEMZP4BSE6FCWBXEVKZL7ESJNNBDNWQMEZDVTQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:54148->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:54148->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->45.91.116.5:4297: use of closed network connection Dec 27 15:05:39.142838484 (527h36m39.013921529s) [noise=true , address=45.91.116.5:4297 , piece_id=KUKOTA75QZZ62NXEMZP4BSE6FCWBXEVKZL7ESJNNBDNWQMEZDVTQ, seg_pos={0 0}, upload=462772]: Long tail cancellation due to optimal threshold Dec 27 15:05:39.143003759 (527h36m39.014086804s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDGQKJEBT34LRQJQVS2LTFQGEUTM2BTE65R3ZSG26E33SCTCSVQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 27 15:05:39.143050798 (527h36m39.014133863s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDGQKJEBT34LRQJQVS2LTFQGEUTM2BTE65R3ZSG26E33SCTCSVQ, seg_pos={0 0}, upload=462772]: Long tail cancellation due to optimal threshold Dec 27 15:05:39.143435481 (527h36m39.014518526s) [noise=true , address=45.91.116.6:4975 , piece_id=U2U4Z2WSMEJ426XIQDR4N5UAVRKCLU57C6FAD7ZP3CFM7CDQBXNA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:57402->45.91.116.6:4975: use of closed network connection; context canceled Dec 27 15:05:39.143445130 (527h36m39.014528165s) [noise=true , address=45.91.116.6:4975 , piece_id=U2U4Z2WSMEJ426XIQDR4N5UAVRKCLU57C6FAD7ZP3CFM7CDQBXNA, seg_pos={0 0}, upload=462772]: Long tail cancellation due to optimal threshold Dec 27 15:05:39.143484234 (527h36m39.014567279s) [seg_pos={0 0}, upload=462772]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:05:39.143578694 (527h36m39.014661729s) [upload=462772]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:05:39.143602579 (527h36m39.014685624s) [upload=462772]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:05:39.439589143 (527h36m39.310672188s) [upload=462772]: Done uploading segments... Dec 27 15:05:39.439612577 (527h36m39.310695622s) [upload=462772]: Upload finished. err: <nil> Duration: 30.004095007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.............X..X...........X..............X....X.......] duration: 44.789389182s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:05:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 44.789403949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 13.309279855s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:05:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 13.309304741s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X] duration: 5.001305553s trace: https://jaeger.dev.storj.io/trace/677729cf7e56cee FAIL (2025-12-27 15:05:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001331262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X..X....X.........X.X....X...X.....X..XX.......X..X.X..X...] duration: 5.792839693s trace: https://jaeger.dev.storj.io/trace/28000095df6a56bb FAIL (2025-12-27 15:05:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded ======LOGS====== ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments... Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment... Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment. Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments... Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces. Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments... Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil> Duration: 5.792871112s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 13.468723265s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 13.468766378s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.............X..X...........X..............X....X......] duration: 45.164921547s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:04:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 45.164941044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X..X....X.........X.X....X...X.....X..XX.......X..X.X..X..] duration: 5.762749226s trace: https://jaeger.dev.storj.io/trace/28000095df6a56bb FAIL (2025-12-27 15:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded ======LOGS====== ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments... Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment... Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment. Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments... Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces. Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments... Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil> Duration: 5.762776386s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.] duration: 4.748570308s trace: https://jaeger.dev.storj.io/trace/1fc947a1373dfe7a FAIL (2025-12-27 15:04:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.748585076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.............X..X...........X..............X....X.....] duration: 39.049451984s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:03:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 39.049478966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X..X....X.........X.X....X...X.....X..XX.......X..X.X..X.] duration: 6.152988402s trace: https://jaeger.dev.storj.io/trace/28000095df6a56bb FAIL (2025-12-27 15:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded ======LOGS====== ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments... Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment... Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment. Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments... Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces. Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments... Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil> Duration: 6.153015564s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX] duration: 5.001228297s trace: https://jaeger.dev.storj.io/trace/1fc947a1373dfe7a FAIL (2025-12-27 15:03:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001252964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 8.977762029s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:03:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 8.977793468s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X..X....X.........X.X....X...X.....X..XX.......X..X.X..X] duration: 30.003987301s trace: https://jaeger.dev.storj.io/trace/28000095df6a56bb FAIL (2025-12-27 15:03:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded ======LOGS====== ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments... Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment... Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment. Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece... Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments... Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil> Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces. Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments... Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil> Duration: 30.004010106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.............X..X...........X..............X....X....] duration: 38.919461848s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:03:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 38.919554515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XX] duration: 5.001544537s trace: https://jaeger.dev.storj.io/trace/35f0c0117051a504 FAIL (2025-12-27 15:03:09 UTC) orbiter warning: context deadline exceeded Duration: 5.001568132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....X] duration: 5.000801425s trace: https://jaeger.dev.storj.io/trace/6d55fbd39c85eaea FAIL (2025-12-27 15:02:45 UTC) orbiter warning: context deadline exceeded Duration: 5.00081431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X..X....X.........X.X....X...X.....X..XX.......X..X.X..] duration: 24.869718596s trace: https://jaeger.dev.storj.io/trace/4e17d619f11f1314 FAIL (2025-12-27 15:02:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ) context deadline exceeded ======LOGS====== [upload=462695]: Done starting upload Dec 27 15:00:26.740248526 (527h31m26.611331581s) [upload=462695]: Uploading segments... Dec 27 15:00:26.740469176 (527h31m26.611552231s) [upload=462695]: Got next segment. Inline: false Dec 27 15:00:26.740530092 (527h31m26.611613137s) [upload=462695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:00:26.740536604 (527h31m26.611619649s) [upload=462695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:00:26.987676360 (527h31m26.858759405s) [seg_pos={0 0}, upload=462695]: Begin upload segment... Dec 27 15:00:26.987745171 (527h31m26.858828216s) [seg_pos={0 0}, upload=462695]: Done begin upload segment. Dec 27 15:00:26.987767023 (527h31m26.858850068s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.987781821 (527h31m26.858864856s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.988483888 (527h31m26.859566933s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.988949946 (527h31m26.860032981s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.989352211 (527h31m26.860435256s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.989788182 (527h31m26.860871227s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.990311528 (527h31m26.861394563s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.990802994 (527h31m26.861886039s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.991329847 (527h31m26.862412892s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.991821062 (527h31m26.862904107s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.992278774 (527h31m26.863361819s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.992734992 (527h31m26.863818037s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.993218153 (527h31m26.864301198s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.993735378 (527h31m26.864818423s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.994259936 (527h31m26.865342981s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.994741253 (527h31m26.865824298s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:28.604826904 (527h31m28.475909949s) [upload=462695]: Next returned nil segment Dec 27 15:00:28.604842684 (527h31m28.475925729s) [upload=462695]: Waiting for error group managing segments... Dec 27 15:00:28.990758164 (527h31m28.861841209s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.000601877 (527h31m28.871684922s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.005564678 (527h31m28.876647723s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.008325564 (527h31m28.87940862s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.012462602 (527h31m28.883545647s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.017141162 (527h31m28.888224207s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.030747159 (527h31m28.901830204s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.257567985 (527h31m29.12865103s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.281334640 (527h31m29.152417705s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.282698127 (527h31m29.153781172s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.286111437 (527h31m29.157194482s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.335636440 (527h31m29.206719475s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.398278622 (527h31m29.269361657s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.398299693 (527h31m29.269382738s) [seg_pos={0 0}, upload=462695]: Segment reached optimal threshold of 13 pieces. Dec 27 15:00:29.398467242 (527h31m29.269550287s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection Dec 27 15:00:29.398473864 (527h31m29.269556899s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398508831 (527h31m29.269591886s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 15:00:29.398524260 (527h31m29.269607305s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398683263 (527h31m29.269766308s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection Dec 27 15:00:29.398689976 (527h31m29.269773021s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398706236 (527h31m29.269789281s) [seg_pos={0 0}, upload=462695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:00:29.398771741 (527h31m29.269854776s) [upload=462695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:00:29.398776370 (527h31m29.269859405s) [upload=462695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:00:29.696725787 (527h31m29.567808842s) [upload=462695]: Done uploading segments... Dec 27 15:00:29.696752318 (527h31m29.567835363s) [upload=462695]: Upload finished. err: <nil> Duration: 24.869747019s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 12.260443136s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:02:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 12.260473924s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....] duration: 5.001693978s trace: https://jaeger.dev.storj.io/trace/42abe1eeecd8795d FAIL (2025-12-27 15:02:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001721148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.............X..X...........X..............X....X...] duration: 18.54656369s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:01:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 18.546581885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 10.135731518s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 15:01:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 10.135756295s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.............X..X...........X..............X....X..] duration: 52.666286196s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 52.666300956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X..X....X.........X.X....X...X.....X..XX.......X..X.X.] duration: 5.949221833s trace: https://jaeger.dev.storj.io/trace/4e17d619f11f1314 FAIL (2025-12-27 15:01:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ) context deadline exceeded ======LOGS====== [upload=462695]: Done starting upload Dec 27 15:00:26.740248526 (527h31m26.611331581s) [upload=462695]: Uploading segments... Dec 27 15:00:26.740469176 (527h31m26.611552231s) [upload=462695]: Got next segment. Inline: false Dec 27 15:00:26.740530092 (527h31m26.611613137s) [upload=462695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:00:26.740536604 (527h31m26.611619649s) [upload=462695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:00:26.987676360 (527h31m26.858759405s) [seg_pos={0 0}, upload=462695]: Begin upload segment... Dec 27 15:00:26.987745171 (527h31m26.858828216s) [seg_pos={0 0}, upload=462695]: Done begin upload segment. Dec 27 15:00:26.987767023 (527h31m26.858850068s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.987781821 (527h31m26.858864856s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.988483888 (527h31m26.859566933s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.988949946 (527h31m26.860032981s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.989352211 (527h31m26.860435256s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.989788182 (527h31m26.860871227s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.990311528 (527h31m26.861394563s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.990802994 (527h31m26.861886039s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.991329847 (527h31m26.862412892s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.991821062 (527h31m26.862904107s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.992278774 (527h31m26.863361819s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.992734992 (527h31m26.863818037s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.993218153 (527h31m26.864301198s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.993735378 (527h31m26.864818423s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.994259936 (527h31m26.865342981s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.994741253 (527h31m26.865824298s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:28.604826904 (527h31m28.475909949s) [upload=462695]: Next returned nil segment Dec 27 15:00:28.604842684 (527h31m28.475925729s) [upload=462695]: Waiting for error group managing segments... Dec 27 15:00:28.990758164 (527h31m28.861841209s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.000601877 (527h31m28.871684922s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.005564678 (527h31m28.876647723s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.008325564 (527h31m28.87940862s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.012462602 (527h31m28.883545647s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.017141162 (527h31m28.888224207s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.030747159 (527h31m28.901830204s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.257567985 (527h31m29.12865103s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.281334640 (527h31m29.152417705s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.282698127 (527h31m29.153781172s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.286111437 (527h31m29.157194482s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.335636440 (527h31m29.206719475s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.398278622 (527h31m29.269361657s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.398299693 (527h31m29.269382738s) [seg_pos={0 0}, upload=462695]: Segment reached optimal threshold of 13 pieces. Dec 27 15:00:29.398467242 (527h31m29.269550287s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection Dec 27 15:00:29.398473864 (527h31m29.269556899s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398508831 (527h31m29.269591886s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 15:00:29.398524260 (527h31m29.269607305s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398683263 (527h31m29.269766308s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection Dec 27 15:00:29.398689976 (527h31m29.269773021s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398706236 (527h31m29.269789281s) [seg_pos={0 0}, upload=462695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:00:29.398771741 (527h31m29.269854776s) [upload=462695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:00:29.398776370 (527h31m29.269859405s) [upload=462695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:00:29.696725787 (527h31m29.567808842s) [upload=462695]: Done uploading segments... Dec 27 15:00:29.696752318 (527h31m29.567835363s) [upload=462695]: Upload finished. err: <nil> Duration: 5.949255627s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX....] duration: 4.781473949s trace: https://jaeger.dev.storj.io/trace/42abe1eeecd8795d FAIL (2025-12-27 15:01:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.781507463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X..X....X.........X.X....X...X.....X..XX.......X..X.X] duration: 30.003625388s trace: https://jaeger.dev.storj.io/trace/4e17d619f11f1314 FAIL (2025-12-27 15:00:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ) context deadline exceeded ======LOGS====== [upload=462695]: Done starting upload Dec 27 15:00:26.740248526 (527h31m26.611331581s) [upload=462695]: Uploading segments... Dec 27 15:00:26.740469176 (527h31m26.611552231s) [upload=462695]: Got next segment. Inline: false Dec 27 15:00:26.740530092 (527h31m26.611613137s) [upload=462695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:00:26.740536604 (527h31m26.611619649s) [upload=462695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:00:26.987676360 (527h31m26.858759405s) [seg_pos={0 0}, upload=462695]: Begin upload segment... Dec 27 15:00:26.987745171 (527h31m26.858828216s) [seg_pos={0 0}, upload=462695]: Done begin upload segment. Dec 27 15:00:26.987767023 (527h31m26.858850068s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.987781821 (527h31m26.858864856s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.988483888 (527h31m26.859566933s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.988949946 (527h31m26.860032981s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.989352211 (527h31m26.860435256s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.989788182 (527h31m26.860871227s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.990311528 (527h31m26.861394563s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.990802994 (527h31m26.861886039s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.991329847 (527h31m26.862412892s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.991821062 (527h31m26.862904107s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.992278774 (527h31m26.863361819s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.992734992 (527h31m26.863818037s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.993218153 (527h31m26.864301198s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.993735378 (527h31m26.864818423s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.994259936 (527h31m26.865342981s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:26.994741253 (527h31m26.865824298s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Uploading piece... Dec 27 15:00:28.604826904 (527h31m28.475909949s) [upload=462695]: Next returned nil segment Dec 27 15:00:28.604842684 (527h31m28.475925729s) [upload=462695]: Waiting for error group managing segments... Dec 27 15:00:28.990758164 (527h31m28.861841209s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.000601877 (527h31m28.871684922s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.005564678 (527h31m28.876647723s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.008325564 (527h31m28.87940862s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.012462602 (527h31m28.883545647s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.017141162 (527h31m28.888224207s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.030747159 (527h31m28.901830204s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.257567985 (527h31m29.12865103s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.281334640 (527h31m29.152417705s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.282698127 (527h31m29.153781172s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.286111437 (527h31m29.157194482s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.335636440 (527h31m29.206719475s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.398278622 (527h31m29.269361657s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil> Dec 27 15:00:29.398299693 (527h31m29.269382738s) [seg_pos={0 0}, upload=462695]: Segment reached optimal threshold of 13 pieces. Dec 27 15:00:29.398467242 (527h31m29.269550287s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection Dec 27 15:00:29.398473864 (527h31m29.269556899s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398508831 (527h31m29.269591886s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 15:00:29.398524260 (527h31m29.269607305s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398683263 (527h31m29.269766308s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection Dec 27 15:00:29.398689976 (527h31m29.269773021s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold Dec 27 15:00:29.398706236 (527h31m29.269789281s) [seg_pos={0 0}, upload=462695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:00:29.398771741 (527h31m29.269854776s) [upload=462695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:00:29.398776370 (527h31m29.269859405s) [upload=462695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:00:29.696725787 (527h31m29.567808842s) [upload=462695]: Done uploading segments... Dec 27 15:00:29.696752318 (527h31m29.567835363s) [upload=462695]: Upload finished. err: <nil> Duration: 30.003645656s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)