Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X....X.........X.X....X...X.....X..XX.......X..X.X..X...XX] duration:
30.005120389s trace:
https://jaeger.dev.storj.io/trace/6497bd92fd125e9d
FAIL (2025-12-27 15:06:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 15:06:06.530207418 (527h37m6.401290463s) [upload=462780]: Starting upload
Dec 27 15:06:06.530297489 (527h37m6.401380534s) [upload=462780]: Done starting upload
Dec 27 15:06:06.530495226 (527h37m6.401578271s) [upload=462780]: Uploading segments...
Dec 27 15:06:06.530630955 (527h37m6.401714s) [upload=462780]: Got next segment. Inline: false
Dec 27 15:06:06.530674227 (527h37m6.401757262s) [upload=462780]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:06:06.530680508 (527h37m6.401763544s) [upload=462780]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:06:06.775815641 (527h37m6.646898696s) [seg_pos={0 0}, upload=462780]: Begin upload segment...
Dec 27 15:06:06.775880515 (527h37m6.64696356s) [seg_pos={0 0}, upload=462780]: Done begin upload segment.
Dec 27 15:06:06.775953414 (527h37m6.647036459s) [noise=true , address=77.90.31.10:4518 , piece_id=MEP2B3VSDAQT3GOISNNVB6NXGH7EK7JDQHFXI7TZ26ZPLCR7MWZA, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.776015412 (527h37m6.647098457s) [noise=true , address=145.79.227.11:4730 , piece_id=YZAKMS3R2WVHWNBKWW5VSR436VR5MJEXMMM563YNNQ7LJFUG5NAQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.776597980 (527h37m6.647681025s) [noise=true , address=145.79.227.40:4051 , piece_id=C4PAW4IXQ3VJX47CPSDOMQN2LLEF33TRR2V2GFUARD3BP6AJE4JA, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.776792972 (527h37m6.647876017s) [noise=true , address=45.91.116.12:4227 , piece_id=VUI7MSJDYJXDYDHEGTQP3FMX3KFKKXKSCTMG375T2DE3B2K2TYSQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.777089187 (527h37m6.648172232s) [noise=true , address=145.79.227.21:4474 , piece_id=GQ7L2H37GUGPQHKUH43IRSF6GXP22JOKNWBTLDAQE6VAXEMZT53A, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.777290639 (527h37m6.648373684s) [noise=true , address=45.91.116.6:4973 , piece_id=D2TNNMKCSNHMULLXG44RVDQOHC2N52TIOMZK4GNB6QDNQNMYXZ6A, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.777540767 (527h37m6.648623812s) [noise=true , address=145.79.227.10:4403 , piece_id=RBHGUGQUK4JOGUOI7AKQFQYPD2YH5BA4GU3AZKH22UDWYYVZ4KDA, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.777780622 (527h37m6.648863657s) [noise=true , address=145.79.227.42:4123 , piece_id=MS425GSZPZ6AJSVJMD7AMLWAL5S6IAP47JPP3FID4WQEGBPUVRDQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.778088729 (527h37m6.649171774s) [noise=true , address=45.91.116.14:4189 , piece_id=D3NRSZOALDMKDCLHRCTBHWHAFN22ESBOE2FNDTIZN2JN4RUENNIA, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.778513549 (527h37m6.649596594s) [noise=true , address=145.79.227.29:4943 , piece_id=YKA5I3SFYHKPJ5ZEYITMTUNXAVYY3D3TFGO2NZCQ3JLC2ZS5LI7A, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.778918209 (527h37m6.650001244s) [noise=true , address=145.79.227.16:4911 , piece_id=5QUTNLNAOMZ4KXUU76TFUVNZYOMIDENM7TESP7OC2BSGBP2432CQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.779349070 (527h37m6.650432115s) [noise=true , address=77.90.31.13:4629 , piece_id=PSHW3JVMSMWKMW524FDTQLWJUOUMRVTSA3Y26SFDXJ2JHZVSIOQQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.779855944 (527h37m6.65093899s) [noise=true , address=145.79.227.5:4221 , piece_id=3QF4D55L7GCR5JRZQ744CX5YPT7N7I4BS5VYXMGTD4ELMFVIINWQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.780232993 (527h37m6.651316028s) [noise=true , address=145.79.227.12:4765 , piece_id=TUB4OOOCXLNIUWVOV3NZLUGG4MMTRQ7J4C4S4UGA7SRTIHFDGTLQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.780797788 (527h37m6.651880833s) [noise=true , address=145.79.227.50:4373 , piece_id=GAL5IAZDMWVD6366WAYWCOBAGKDBKALWEPD3HOKKV634664WJB5Q, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:06.781200966 (527h37m6.652284011s) [noise=true , address=145.79.227.17:4971 , piece_id=MWLSPGQ34AL767LHKKFTHPDOESBEIPOC7NBTUCRKY5XF4B5GVDAQ, seg_pos={0 0}, upload=462780]: Uploading piece...
Dec 27 15:06:08.391150188 (527h37m8.262233223s) [upload=462780]: Next returned nil segment
Dec 27 15:06:08.391170056 (527h37m8.262253101s) [upload=462780]: Waiting for error group managing segments...
Dec 27 15:06:08.779769521 (527h37m8.650852566s) [noise=true , address=145.79.227.16:4911 , piece_id=5QUTNLNAOMZ4KXUU76TFUVNZYOMIDENM7TESP7OC2BSGBP2432CQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.786709607 (527h37m8.657792652s) [noise=true , address=145.79.227.17:4971 , piece_id=MWLSPGQ34AL767LHKKFTHPDOESBEIPOC7NBTUCRKY5XF4B5GVDAQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.786924947 (527h37m8.658007992s) [noise=true , address=145.79.227.11:4730 , piece_id=YZAKMS3R2WVHWNBKWW5VSR436VR5MJEXMMM563YNNQ7LJFUG5NAQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.787929931 (527h37m8.659012976s) [noise=true , address=145.79.227.21:4474 , piece_id=GQ7L2H37GUGPQHKUH43IRSF6GXP22JOKNWBTLDAQE6VAXEMZT53A, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.790941995 (527h37m8.662025031s) [noise=true , address=145.79.227.42:4123 , piece_id=MS425GSZPZ6AJSVJMD7AMLWAL5S6IAP47JPP3FID4WQEGBPUVRDQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.791999730 (527h37m8.663082775s) [noise=true , address=145.79.227.29:4943 , piece_id=YKA5I3SFYHKPJ5ZEYITMTUNXAVYY3D3TFGO2NZCQ3JLC2ZS5LI7A, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.803024974 (527h37m8.674108009s) [noise=true , address=145.79.227.5:4221 , piece_id=3QF4D55L7GCR5JRZQ744CX5YPT7N7I4BS5VYXMGTD4ELMFVIINWQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.803151055 (527h37m8.67423409s) [noise=true , address=145.79.227.10:4403 , piece_id=RBHGUGQUK4JOGUOI7AKQFQYPD2YH5BA4GU3AZKH22UDWYYVZ4KDA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.805604276 (527h37m8.676687311s) [noise=true , address=145.79.227.50:4373 , piece_id=GAL5IAZDMWVD6366WAYWCOBAGKDBKALWEPD3HOKKV634664WJB5Q, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:08.809331353 (527h37m8.680414398s) [noise=true , address=145.79.227.12:4765 , piece_id=TUB4OOOCXLNIUWVOV3NZLUGG4MMTRQ7J4C4S4UGA7SRTIHFDGTLQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:09.041305505 (527h37m8.91238854s) [noise=true , address=77.90.31.10:4518 , piece_id=MEP2B3VSDAQT3GOISNNVB6NXGH7EK7JDQHFXI7TZ26ZPLCR7MWZA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:09.041878486 (527h37m8.912961531s) [noise=true , address=45.91.116.6:4973 , piece_id=D2TNNMKCSNHMULLXG44RVDQOHC2N52TIOMZK4GNB6QDNQNMYXZ6A, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:09.572028099 (527h37m9.443111144s) [noise=true , address=45.91.116.14:4189 , piece_id=D3NRSZOALDMKDCLHRCTBHWHAFN22ESBOE2FNDTIZN2JN4RUENNIA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: <nil>
Dec 27 15:06:09.572048749 (527h37m9.443131794s) [seg_pos={0 0}, upload=462780]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:06:09.572261314 (527h37m9.443344349s) [noise=true , address=77.90.31.13:4629 , piece_id=PSHW3JVMSMWKMW524FDTQLWJUOUMRVTSA3Y26SFDXJ2JHZVSIOQQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF
Dec 27 15:06:09.572267235 (527h37m9.44335028s) [noise=true , address=77.90.31.13:4629 , piece_id=PSHW3JVMSMWKMW524FDTQLWJUOUMRVTSA3Y26SFDXJ2JHZVSIOQQ, seg_pos={0 0}, upload=462780]: Long tail cancellation due to optimal threshold
Dec 27 15:06:09.572503635 (527h37m9.44358668s) [noise=true , address=45.91.116.12:4227 , piece_id=VUI7MSJDYJXDYDHEGTQP3FMX3KFKKXKSCTMG375T2DE3B2K2TYSQ, seg_pos={0 0}, upload=462780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:56564->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:56564->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56564->45.91.116.12:4227: use of closed network connection
Dec 27 15:06:09.572513564 (527h37m9.443596609s) [noise=true , address=45.91.116.12:4227 , piece_id=VUI7MSJDYJXDYDHEGTQP3FMX3KFKKXKSCTMG375T2DE3B2K2TYSQ, seg_pos={0 0}, upload=462780]: Long tail cancellation due to optimal threshold
Dec 27 15:06:09.572708013 (527h37m9.443791078s) [noise=true , address=145.79.227.40:4051 , piece_id=C4PAW4IXQ3VJX47CPSDOMQN2LLEF33TRR2V2GFUARD3BP6AJE4JA, seg_pos={0 0}, upload=462780]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 27 15:06:09.572747479 (527h37m9.443830544s) [noise=true , address=145.79.227.40:4051 , piece_id=C4PAW4IXQ3VJX47CPSDOMQN2LLEF33TRR2V2GFUARD3BP6AJE4JA, seg_pos={0 0}, upload=462780]: Long tail cancellation due to optimal threshold
Dec 27 15:06:09.572779430 (527h37m9.443862475s) [seg_pos={0 0}, upload=462780]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:06:09.572914717 (527h37m9.443997762s) [upload=462780]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:06:09.572922412 (527h37m9.444005457s) [upload=462780]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:06:09.863899917 (527h37m9.734982962s) [upload=462780]: Done uploading segments...
Dec 27 15:06:09.863923492 (527h37m9.735006537s) [upload=462780]: Upload finished. err: <nil>
Duration: 30.00513657s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
10.770482278s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 15:06:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 10.770511633s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X.............X..X...........X..............X....X........] duration:
29.892838803s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:06:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 29.892863679s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [XX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.] duration:
3.926139969s trace:
https://jaeger.dev.storj.io/trace/677729cf7e56cee
FAIL (2025-12-27 15:06:14 UTC)
orbiter warning: context deadline exceeded
Duration: 3.926168522s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X..X....X.........X.X....X...X.....X..XX.......X..X.X..X...X] duration:
30.004069318s trace:
https://jaeger.dev.storj.io/trace/2d2d647d95d7cd9c
FAIL (2025-12-27 15:06:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SL7HZGRJRIKILJKXYP3KNJPSR5UV25IL26VQV5X44QAKHDNSKRAQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 6DX76LZSTIIAZNYYUOGOWJVGYIQBNXAS5QR6EOSFQSQDPE6N5JQA) context deadline exceeded
======LOGS======
72]: Starting upload
Dec 27 15:05:36.276416804 (527h36m36.147499839s) [upload=462772]: Done starting upload
Dec 27 15:05:36.276486918 (527h36m36.147569953s) [upload=462772]: Uploading segments...
Dec 27 15:05:36.276584193 (527h36m36.147667228s) [upload=462772]: Got next segment. Inline: false
Dec 27 15:05:36.276628728 (527h36m36.147711763s) [upload=462772]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:05:36.276634228 (527h36m36.147717273s) [upload=462772]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:05:36.531893098 (527h36m36.402976153s) [seg_pos={0 0}, upload=462772]: Begin upload segment...
Dec 27 15:05:36.531948665 (527h36m36.40303174s) [seg_pos={0 0}, upload=462772]: Done begin upload segment.
Dec 27 15:05:36.532033796 (527h36m36.403116841s) [noise=true , address=145.79.227.39:4015 , piece_id=ZXMRGWDE4ZTFNMLS5BH2STTPBT3JMI4HULEQ3AUSEMHMDXTQOLDQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.532091036 (527h36m36.403174081s) [noise=true , address=145.79.227.48:4302 , piece_id=ASRGOO4LXNGVYNSJNT7XG7NUBAUNBV4SJT3RRGIA6M6JUO3MTH4A, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.532603941 (527h36m36.403686977s) [noise=true , address=45.91.116.8:4047 , piece_id=6DX76LZSTIIAZNYYUOGOWJVGYIQBNXAS5QR6EOSFQSQDPE6N5JQA, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.532651813 (527h36m36.403734848s) [noise=true , address=145.79.227.41:4085 , piece_id=4MHSTWNILDWAGBH4IEJX6SP5BQGV722R7S4XI2JFICINLG3HXTNQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.533010065 (527h36m36.4040931s) [noise=true , address=45.91.116.5:4297 , piece_id=KUKOTA75QZZ62NXEMZP4BSE6FCWBXEVKZL7ESJNNBDNWQMEZDVTQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.533632360 (527h36m36.404715405s) [noise=true , address=145.79.227.42:4122 , piece_id=C7KEWEPFJ2DVJMOACPOWTOAPPEOVFN2W5WF5LDRQX6O6JNN5BO3A, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.534261829 (527h36m36.405344864s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDGQKJEBT34LRQJQVS2LTFQGEUTM2BTE65R3ZSG26E33SCTCSVQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.534550869 (527h36m36.405633904s) [noise=true , address=145.79.227.10:4403 , piece_id=7N4MD6PL3KDB5VUF77QHSLSUUH4XSEG72RNKEGJKJGMTHYSZ7JWQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.535141293 (527h36m36.406224338s) [noise=true , address=145.79.227.21:4475 , piece_id=NEGHPLVNG75KAGVSCE6KAN735G4QWNKAAXUR3YP7PDLOFD2PYUGA, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.535403964 (527h36m36.406487009s) [noise=true , address=145.79.227.45:4229 , piece_id=23KSB6OFP5FGVQCRZXTT3YRZBTYVXFBWID2XAIOZJNWYQKYY24XA, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.535985501 (527h36m36.407068546s) [noise=true , address=77.90.31.21:4880 , piece_id=SL7HZGRJRIKILJKXYP3KNJPSR5UV25IL26VQV5X44QAKHDNSKRAQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.536266376 (527h36m36.407349411s) [noise=true , address=145.79.227.49:4339 , piece_id=F4K3FJO5H4V2PNRMSM5VRBG5JORGGV67Y2API6GAUGPRB2OWZOAA, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.536809870 (527h36m36.407892916s) [noise=true , address=145.79.227.47:4266 , piece_id=LP5M4RG56HTSEDHTHXS3TCN2KCEKCSI3MDZKD6RJ5SGSHHSMEK6Q, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.537171620 (527h36m36.408254665s) [noise=true , address=45.91.116.13:4156 , piece_id=WHB2LN2BM7ZRSTYZJIZ6GRIS3UGRSDD4SJUULO4LGVB7Z2OYWWTQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.537477792 (527h36m36.408560827s) [noise=true , address=45.91.116.6:4975 , piece_id=U2U4Z2WSMEJ426XIQDR4N5UAVRKCLU57C6FAD7ZP3CFM7CDQBXNA, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:36.537956093 (527h36m36.409039138s) [noise=true , address=145.79.227.37:4565 , piece_id=JTCAIUEG6H3GVZGJMHTX6Z3Y34L7WU3O7XJPWGF26AFUQIAMRDPQ, seg_pos={0 0}, upload=462772]: Uploading piece...
Dec 27 15:05:38.158872068 (527h36m38.029955113s) [upload=462772]: Next returned nil segment
Dec 27 15:05:38.158888891 (527h36m38.029971926s) [upload=462772]: Waiting for error group managing segments...
Dec 27 15:05:38.543650418 (527h36m38.414733513s) [noise=true , address=145.79.227.39:4015 , piece_id=ZXMRGWDE4ZTFNMLS5BH2STTPBT3JMI4HULEQ3AUSEMHMDXTQOLDQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.546844460 (527h36m38.417927506s) [noise=true , address=145.79.227.45:4229 , piece_id=23KSB6OFP5FGVQCRZXTT3YRZBTYVXFBWID2XAIOZJNWYQKYY24XA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.548801847 (527h36m38.419884883s) [noise=true , address=145.79.227.48:4302 , piece_id=ASRGOO4LXNGVYNSJNT7XG7NUBAUNBV4SJT3RRGIA6M6JUO3MTH4A, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.549236926 (527h36m38.420319971s) [noise=true , address=145.79.227.37:4565 , piece_id=JTCAIUEG6H3GVZGJMHTX6Z3Y34L7WU3O7XJPWGF26AFUQIAMRDPQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.549324974 (527h36m38.420408019s) [noise=true , address=145.79.227.42:4122 , piece_id=C7KEWEPFJ2DVJMOACPOWTOAPPEOVFN2W5WF5LDRQX6O6JNN5BO3A, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.550006161 (527h36m38.421089196s) [noise=true , address=145.79.227.41:4085 , piece_id=4MHSTWNILDWAGBH4IEJX6SP5BQGV722R7S4XI2JFICINLG3HXTNQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.558655781 (527h36m38.429738816s) [noise=true , address=145.79.227.21:4475 , piece_id=NEGHPLVNG75KAGVSCE6KAN735G4QWNKAAXUR3YP7PDLOFD2PYUGA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.562459023 (527h36m38.433542068s) [noise=true , address=145.79.227.10:4403 , piece_id=7N4MD6PL3KDB5VUF77QHSLSUUH4XSEG72RNKEGJKJGMTHYSZ7JWQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.572453234 (527h36m38.443536279s) [noise=true , address=145.79.227.49:4339 , piece_id=F4K3FJO5H4V2PNRMSM5VRBG5JORGGV67Y2API6GAUGPRB2OWZOAA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.575805217 (527h36m38.446888262s) [noise=true , address=145.79.227.47:4266 , piece_id=LP5M4RG56HTSEDHTHXS3TCN2KCEKCSI3MDZKD6RJ5SGSHHSMEK6Q, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.815643705 (527h36m38.68672674s) [noise=true , address=77.90.31.21:4880 , piece_id=SL7HZGRJRIKILJKXYP3KNJPSR5UV25IL26VQV5X44QAKHDNSKRAQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:38.945441103 (527h36m38.816524148s) [noise=true , address=45.91.116.13:4156 , piece_id=WHB2LN2BM7ZRSTYZJIZ6GRIS3UGRSDD4SJUULO4LGVB7Z2OYWWTQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:39.142513235 (527h36m39.01359627s) [noise=true , address=45.91.116.8:4047 , piece_id=6DX76LZSTIIAZNYYUOGOWJVGYIQBNXAS5QR6EOSFQSQDPE6N5JQA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: <nil>
Dec 27 15:05:39.142539615 (527h36m39.013622661s) [seg_pos={0 0}, upload=462772]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:05:39.142828846 (527h36m39.013911901s) [noise=true , address=45.91.116.5:4297 , piece_id=KUKOTA75QZZ62NXEMZP4BSE6FCWBXEVKZL7ESJNNBDNWQMEZDVTQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:54148->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:54148->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54148->45.91.116.5:4297: use of closed network connection
Dec 27 15:05:39.142838484 (527h36m39.013921529s) [noise=true , address=45.91.116.5:4297 , piece_id=KUKOTA75QZZ62NXEMZP4BSE6FCWBXEVKZL7ESJNNBDNWQMEZDVTQ, seg_pos={0 0}, upload=462772]: Long tail cancellation due to optimal threshold
Dec 27 15:05:39.143003759 (527h36m39.014086804s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDGQKJEBT34LRQJQVS2LTFQGEUTM2BTE65R3ZSG26E33SCTCSVQ, seg_pos={0 0}, upload=462772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 15:05:39.143050798 (527h36m39.014133863s) [noise=true , address=77.90.31.16:4735 , piece_id=ZFDGQKJEBT34LRQJQVS2LTFQGEUTM2BTE65R3ZSG26E33SCTCSVQ, seg_pos={0 0}, upload=462772]: Long tail cancellation due to optimal threshold
Dec 27 15:05:39.143435481 (527h36m39.014518526s) [noise=true , address=45.91.116.6:4975 , piece_id=U2U4Z2WSMEJ426XIQDR4N5UAVRKCLU57C6FAD7ZP3CFM7CDQBXNA, seg_pos={0 0}, upload=462772]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:57402->45.91.116.6:4975: use of closed network connection; context canceled
Dec 27 15:05:39.143445130 (527h36m39.014528165s) [noise=true , address=45.91.116.6:4975 , piece_id=U2U4Z2WSMEJ426XIQDR4N5UAVRKCLU57C6FAD7ZP3CFM7CDQBXNA, seg_pos={0 0}, upload=462772]: Long tail cancellation due to optimal threshold
Dec 27 15:05:39.143484234 (527h36m39.014567279s) [seg_pos={0 0}, upload=462772]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:05:39.143578694 (527h36m39.014661729s) [upload=462772]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:05:39.143602579 (527h36m39.014685624s) [upload=462772]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:05:39.439589143 (527h36m39.310672188s) [upload=462772]: Done uploading segments...
Dec 27 15:05:39.439612577 (527h36m39.310695622s) [upload=462772]: Upload finished. err: <nil>
Duration: 30.004095007s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X.............X..X...........X..............X....X.......] duration:
44.789389182s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:05:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 44.789403949s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
13.309279855s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 15:05:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 13.309304741s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X] duration:
5.001305553s trace:
https://jaeger.dev.storj.io/trace/677729cf7e56cee
FAIL (2025-12-27 15:05:15 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001331262s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X..X....X.........X.X....X...X.....X..XX.......X..X.X..X...] duration:
5.792839693s trace:
https://jaeger.dev.storj.io/trace/28000095df6a56bb
FAIL (2025-12-27 15:05:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded
======LOGS======
ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload
Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload
Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments...
Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false
Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment...
Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment.
Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment
Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments...
Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection
Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments...
Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil>
Duration: 5.792871112s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
13.468723265s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 15:04:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 13.468766378s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X.............X..X...........X..............X....X......] duration:
45.164921547s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:04:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 45.164941044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X..X....X.........X.X....X...X.....X..XX.......X..X.X..X..] duration:
5.762749226s trace:
https://jaeger.dev.storj.io/trace/28000095df6a56bb
FAIL (2025-12-27 15:04:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded
======LOGS======
ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload
Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload
Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments...
Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false
Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment...
Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment.
Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment
Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments...
Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection
Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments...
Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil>
Duration: 5.762776386s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.] duration:
4.748570308s trace:
https://jaeger.dev.storj.io/trace/1fc947a1373dfe7a
FAIL (2025-12-27 15:04:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.748585076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X.............X..X...........X..............X....X.....] duration:
39.049451984s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:03:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 39.049478966s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X..X....X.........X.X....X...X.....X..XX.......X..X.X..X.] duration:
6.152988402s trace:
https://jaeger.dev.storj.io/trace/28000095df6a56bb
FAIL (2025-12-27 15:03:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded
======LOGS======
ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload
Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload
Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments...
Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false
Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment...
Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment.
Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment
Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments...
Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection
Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments...
Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil>
Duration: 6.153015564s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [XXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX] duration:
5.001228297s trace:
https://jaeger.dev.storj.io/trace/1fc947a1373dfe7a
FAIL (2025-12-27 15:03:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001252964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
8.977762029s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 15:03:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 8.977793468s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X..X....X.........X.X....X...X.....X..XX.......X..X.X..X] duration:
30.003987301s trace:
https://jaeger.dev.storj.io/trace/28000095df6a56bb
FAIL (2025-12-27 15:03:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA) context deadline exceeded
======LOGS======
ec 27 15:02:52.105942881 (527h33m51.977025926s) [upload=462733]: Starting upload
Dec 27 15:02:52.106024296 (527h33m51.977107351s) [upload=462733]: Done starting upload
Dec 27 15:02:52.106098929 (527h33m51.977181974s) [upload=462733]: Uploading segments...
Dec 27 15:02:52.106266738 (527h33m51.977349783s) [upload=462733]: Got next segment. Inline: false
Dec 27 15:02:52.106312035 (527h33m51.97739507s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:02:52.106317244 (527h33m51.97740029s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:02:52.358470057 (527h33m52.229553113s) [seg_pos={0 0}, upload=462733]: Begin upload segment...
Dec 27 15:02:52.358524771 (527h33m52.229607806s) [seg_pos={0 0}, upload=462733]: Done begin upload segment.
Dec 27 15:02:52.358552835 (527h33m52.22963588s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.358616947 (527h33m52.229699982s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359065581 (527h33m52.230148626s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.359797014 (527h33m52.230880049s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360499261 (527h33m52.231582296s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.360834530 (527h33m52.231917565s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.361231245 (527h33m52.23231428s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.362718357 (527h33m52.233801392s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.363001496 (527h33m52.234084541s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364203325 (527h33m52.23528636s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.364895332 (527h33m52.235978377s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.366838292 (527h33m52.237921337s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.368509384 (527h33m52.239592429s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.370100143 (527h33m52.241183188s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.371867048 (527h33m52.242950093s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:52.373343510 (527h33m52.244426555s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Uploading piece...
Dec 27 15:02:54.069649902 (527h33m53.940732947s) [upload=462733]: Next returned nil segment
Dec 27 15:02:54.069661143 (527h33m53.940744178s) [upload=462733]: Waiting for error group managing segments...
Dec 27 15:02:54.358283022 (527h33m54.229366067s) [noise=true , address=145.79.227.45:4231 , piece_id=JMCYSG77LXYSJSXEDIY2QA4GYZCKYSOUPR4VC6VURL2EZEMH2R2Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.363794388 (527h33m54.234877423s) [noise=true , address=145.79.227.16:4910 , piece_id=S2ZXS4FT7GSWY7QVFAAC4D5HY43PWYAKBO2CMYQXCVHBZT7N6B6Q, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.380152216 (527h33m54.251235261s) [noise=true , address=145.79.227.36:4518 , piece_id=J65HQXH5CQ54EQTLCDGCIYQMFY5HC2RD3M24EZKVS4SOONEZWZPA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.390607334 (527h33m54.261690389s) [noise=true , address=145.79.227.32:4835 , piece_id=PZKIV5WK3JMEDHXGSVZUXCDQFICZQTLQ36RALPO5SIOXUTHMB5RA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.391155287 (527h33m54.262238323s) [noise=true , address=145.79.227.41:4087 , piece_id=OTAA3BJJ2GKLP2Z3UZMEOAAFJEJ4Z3B5EKYP3EYXNL36FKW4YK7A, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.400264814 (527h33m54.271347849s) [noise=true , address=145.79.227.10:4403 , piece_id=RRUMTYKCGYKZFMWWMNPG5RD3IILBEBRKAT7H46SFRGMVFEMGUSBA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.406683977 (527h33m54.277767032s) [noise=true , address=145.79.227.35:4481 , piece_id=KUKHIKSMI7U646AUXWR7LTROXYXG36TNCSBZ2KP2W6SJK6N65SFA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.407611133 (527h33m54.278694178s) [noise=true , address=145.79.227.9:4365 , piece_id=5TJTS45WA4AW7MY76PKSCCPC4AXAHN56JP6QKIX3RZ4UQPXZKPBQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:54.603981799 (527h33m54.475064844s) [noise=true , address=45.91.116.11:4117 , piece_id=PKY4PJ2DCLY45TEJH72BRWNTURU4MZB5VJDW3ID47IKQRESJV7OA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.303772720 (527h33m55.174855765s) [noise=true , address=45.91.116.5:4298 , piece_id=NB75GT3M65WWGGTBIQBS72A7SOOK2X7KDKCRWQUOPMVWEEHHEVSA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.347988014 (527h33m55.21907106s) [noise=true , address=77.90.31.10:4519 , piece_id=BKMXF6DDE2RV3BVMPZXGCIPHLZYLWIVGKKPOOBTWPLLXJRYARWTQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:55.506895093 (527h33m55.377978128s) [noise=true , address=45.91.116.6:4976 , piece_id=UTNPPPW5EJRD5N4MXABMXBAM2GC7I35AKATIKIFDXTR6ENKDGDRQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002186170 (527h33m55.873269215s) [noise=true , address=77.90.31.16:4736 , piece_id=ODD6CL73VFC3GO4AA2OMKNCA7ASMJAVTOK4MWYNDOQHWFOTQKVQA, seg_pos={0 0}, upload=462733]: Done uploading piece. err: <nil>
Dec 27 15:02:56.002216426 (527h33m55.873299471s) [seg_pos={0 0}, upload=462733]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:02:56.002541767 (527h33m55.873624812s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002553228 (527h33m55.873636273s) [noise=true , address=77.90.31.21:4880 , piece_id=XMMOUKMCX6BIYBB7W6IKNET25XRJPZSSW6KP6OKQ34ELMC4ZG7XQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002796531 (527h33m55.873879576s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51246->45.91.116.7:4011: use of closed network connection
Dec 27 15:02:56.002841256 (527h33m55.873924321s) [noise=true , address=45.91.116.7:4011 , piece_id=7RIE4VJCSBDBXSCZTZPYTFBFQH6UTKNTW6PYWIZNQMWLII6VBKCQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.002888868 (527h33m55.873971913s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 27 15:02:56.002900429 (527h33m55.873983474s) [noise=true , address=45.91.116.9:4262 , piece_id=G52ES77TKCCQODBR26M4PJRRRBM5TSBOHQS627ESRKA7QNC7ZKMQ, seg_pos={0 0}, upload=462733]: Long tail cancellation due to optimal threshold
Dec 27 15:02:56.003053240 (527h33m55.874136275s) [seg_pos={0 0}, upload=462733]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:02:56.003188768 (527h33m55.874271813s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:02:56.003196033 (527h33m55.874279078s) [upload=462733]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:02:56.308597704 (527h33m56.179680739s) [upload=462733]: Done uploading segments...
Dec 27 15:02:56.308613263 (527h33m56.179696298s) [upload=462733]: Upload finished. err: <nil>
Duration: 30.004010106s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X.............X..X...........X..............X....X....] duration:
38.919461848s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:03:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 38.919554515s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XX] duration:
5.001544537s trace:
https://jaeger.dev.storj.io/trace/35f0c0117051a504
FAIL (2025-12-27 15:03:09 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001568132s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....X] duration:
5.000801425s trace:
https://jaeger.dev.storj.io/trace/6d55fbd39c85eaea
FAIL (2025-12-27 15:02:45 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00081431s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.....X..X....X.........X.X....X...X.....X..XX.......X..X.X..] duration:
24.869718596s trace:
https://jaeger.dev.storj.io/trace/4e17d619f11f1314
FAIL (2025-12-27 15:02:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ) context deadline exceeded
======LOGS======
[upload=462695]: Done starting upload
Dec 27 15:00:26.740248526 (527h31m26.611331581s) [upload=462695]: Uploading segments...
Dec 27 15:00:26.740469176 (527h31m26.611552231s) [upload=462695]: Got next segment. Inline: false
Dec 27 15:00:26.740530092 (527h31m26.611613137s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:00:26.740536604 (527h31m26.611619649s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:00:26.987676360 (527h31m26.858759405s) [seg_pos={0 0}, upload=462695]: Begin upload segment...
Dec 27 15:00:26.987745171 (527h31m26.858828216s) [seg_pos={0 0}, upload=462695]: Done begin upload segment.
Dec 27 15:00:26.987767023 (527h31m26.858850068s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.987781821 (527h31m26.858864856s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.988483888 (527h31m26.859566933s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.988949946 (527h31m26.860032981s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.989352211 (527h31m26.860435256s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.989788182 (527h31m26.860871227s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.990311528 (527h31m26.861394563s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.990802994 (527h31m26.861886039s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.991329847 (527h31m26.862412892s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.991821062 (527h31m26.862904107s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.992278774 (527h31m26.863361819s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.992734992 (527h31m26.863818037s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.993218153 (527h31m26.864301198s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.993735378 (527h31m26.864818423s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.994259936 (527h31m26.865342981s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.994741253 (527h31m26.865824298s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:28.604826904 (527h31m28.475909949s) [upload=462695]: Next returned nil segment
Dec 27 15:00:28.604842684 (527h31m28.475925729s) [upload=462695]: Waiting for error group managing segments...
Dec 27 15:00:28.990758164 (527h31m28.861841209s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.000601877 (527h31m28.871684922s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.005564678 (527h31m28.876647723s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.008325564 (527h31m28.87940862s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.012462602 (527h31m28.883545647s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.017141162 (527h31m28.888224207s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.030747159 (527h31m28.901830204s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.257567985 (527h31m29.12865103s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.281334640 (527h31m29.152417705s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.282698127 (527h31m29.153781172s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.286111437 (527h31m29.157194482s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.335636440 (527h31m29.206719475s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.398278622 (527h31m29.269361657s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.398299693 (527h31m29.269382738s) [seg_pos={0 0}, upload=462695]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:00:29.398467242 (527h31m29.269550287s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection
Dec 27 15:00:29.398473864 (527h31m29.269556899s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398508831 (527h31m29.269591886s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 27 15:00:29.398524260 (527h31m29.269607305s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398683263 (527h31m29.269766308s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection
Dec 27 15:00:29.398689976 (527h31m29.269773021s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398706236 (527h31m29.269789281s) [seg_pos={0 0}, upload=462695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:00:29.398771741 (527h31m29.269854776s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:00:29.398776370 (527h31m29.269859405s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:00:29.696725787 (527h31m29.567808842s) [upload=462695]: Done uploading segments...
Dec 27 15:00:29.696752318 (527h31m29.567835363s) [upload=462695]: Upload finished. err: <nil>
Duration: 24.869747019s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
12.260443136s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 15:02:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 12.260473924s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [X.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....] duration:
5.001693978s trace:
https://jaeger.dev.storj.io/trace/42abe1eeecd8795d
FAIL (2025-12-27 15:02:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001721148s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......X.............X..X...........X..............X....X...] duration:
18.54656369s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:01:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 18.546581885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
10.135731518s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 15:01:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 10.135756295s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [........X.............X..X...........X..............X....X..] duration:
52.666286196s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:01:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 52.666300956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [......X..X....X.........X.X....X...X.....X..XX.......X..X.X.] duration:
5.949221833s trace:
https://jaeger.dev.storj.io/trace/4e17d619f11f1314
FAIL (2025-12-27 15:01:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ) context deadline exceeded
======LOGS======
[upload=462695]: Done starting upload
Dec 27 15:00:26.740248526 (527h31m26.611331581s) [upload=462695]: Uploading segments...
Dec 27 15:00:26.740469176 (527h31m26.611552231s) [upload=462695]: Got next segment. Inline: false
Dec 27 15:00:26.740530092 (527h31m26.611613137s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:00:26.740536604 (527h31m26.611619649s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:00:26.987676360 (527h31m26.858759405s) [seg_pos={0 0}, upload=462695]: Begin upload segment...
Dec 27 15:00:26.987745171 (527h31m26.858828216s) [seg_pos={0 0}, upload=462695]: Done begin upload segment.
Dec 27 15:00:26.987767023 (527h31m26.858850068s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.987781821 (527h31m26.858864856s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.988483888 (527h31m26.859566933s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.988949946 (527h31m26.860032981s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.989352211 (527h31m26.860435256s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.989788182 (527h31m26.860871227s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.990311528 (527h31m26.861394563s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.990802994 (527h31m26.861886039s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.991329847 (527h31m26.862412892s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.991821062 (527h31m26.862904107s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.992278774 (527h31m26.863361819s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.992734992 (527h31m26.863818037s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.993218153 (527h31m26.864301198s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.993735378 (527h31m26.864818423s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.994259936 (527h31m26.865342981s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.994741253 (527h31m26.865824298s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:28.604826904 (527h31m28.475909949s) [upload=462695]: Next returned nil segment
Dec 27 15:00:28.604842684 (527h31m28.475925729s) [upload=462695]: Waiting for error group managing segments...
Dec 27 15:00:28.990758164 (527h31m28.861841209s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.000601877 (527h31m28.871684922s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.005564678 (527h31m28.876647723s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.008325564 (527h31m28.87940862s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.012462602 (527h31m28.883545647s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.017141162 (527h31m28.888224207s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.030747159 (527h31m28.901830204s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.257567985 (527h31m29.12865103s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.281334640 (527h31m29.152417705s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.282698127 (527h31m29.153781172s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.286111437 (527h31m29.157194482s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.335636440 (527h31m29.206719475s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.398278622 (527h31m29.269361657s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.398299693 (527h31m29.269382738s) [seg_pos={0 0}, upload=462695]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:00:29.398467242 (527h31m29.269550287s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection
Dec 27 15:00:29.398473864 (527h31m29.269556899s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398508831 (527h31m29.269591886s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 27 15:00:29.398524260 (527h31m29.269607305s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398683263 (527h31m29.269766308s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection
Dec 27 15:00:29.398689976 (527h31m29.269773021s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398706236 (527h31m29.269789281s) [seg_pos={0 0}, upload=462695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:00:29.398771741 (527h31m29.269854776s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:00:29.398776370 (527h31m29.269859405s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:00:29.696725787 (527h31m29.567808842s) [upload=462695]: Done uploading segments...
Dec 27 15:00:29.696752318 (527h31m29.567835363s) [upload=462695]: Upload finished. err: <nil>
Duration: 5.949255627s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX....] duration:
4.781473949s trace:
https://jaeger.dev.storj.io/trace/42abe1eeecd8795d
FAIL (2025-12-27 15:01:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.781507463s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.......X..X....X.........X.X....X...X.....X..XX.......X..X.X] duration:
30.003625388s trace:
https://jaeger.dev.storj.io/trace/4e17d619f11f1314
FAIL (2025-12-27 15:00:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ) context deadline exceeded
======LOGS======
[upload=462695]: Done starting upload
Dec 27 15:00:26.740248526 (527h31m26.611331581s) [upload=462695]: Uploading segments...
Dec 27 15:00:26.740469176 (527h31m26.611552231s) [upload=462695]: Got next segment. Inline: false
Dec 27 15:00:26.740530092 (527h31m26.611613137s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:00:26.740536604 (527h31m26.611619649s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:00:26.987676360 (527h31m26.858759405s) [seg_pos={0 0}, upload=462695]: Begin upload segment...
Dec 27 15:00:26.987745171 (527h31m26.858828216s) [seg_pos={0 0}, upload=462695]: Done begin upload segment.
Dec 27 15:00:26.987767023 (527h31m26.858850068s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.987781821 (527h31m26.858864856s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.988483888 (527h31m26.859566933s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.988949946 (527h31m26.860032981s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.989352211 (527h31m26.860435256s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.989788182 (527h31m26.860871227s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.990311528 (527h31m26.861394563s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.990802994 (527h31m26.861886039s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.991329847 (527h31m26.862412892s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.991821062 (527h31m26.862904107s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.992278774 (527h31m26.863361819s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.992734992 (527h31m26.863818037s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.993218153 (527h31m26.864301198s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.993735378 (527h31m26.864818423s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.994259936 (527h31m26.865342981s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:26.994741253 (527h31m26.865824298s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Uploading piece...
Dec 27 15:00:28.604826904 (527h31m28.475909949s) [upload=462695]: Next returned nil segment
Dec 27 15:00:28.604842684 (527h31m28.475925729s) [upload=462695]: Waiting for error group managing segments...
Dec 27 15:00:28.990758164 (527h31m28.861841209s) [noise=true , address=145.79.227.12:4767 , piece_id=L2KWJZKHSMPUF5RSTMRTEFTPV4VZD6SBT7MEGETGESCVKDYBFTWA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.000601877 (527h31m28.871684922s) [noise=true , address=145.79.227.50:4374 , piece_id=C3DXSERVF75HN6A35RW7RDYL7ZLCLZXWE3I3TOLZA7LWZWJWQ6KQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.005564678 (527h31m28.876647723s) [noise=true , address=145.79.227.4:4186 , piece_id=5TJPBRUOURLCHTW466CPH7UPX6OA7HXSIG7U3BOCNYJ3DOBI5B2Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.008325564 (527h31m28.87940862s) [noise=true , address=145.79.227.45:4230 , piece_id=ID4RIZYD7CZH6T27TKHJ5362AKKNY5PIRSEOZEA7V7C7B2TJSIXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.012462602 (527h31m28.883545647s) [noise=true , address=145.79.227.37:4567 , piece_id=SA2ZC6NPLKD46BQ3XK7EAWJRVYRIYHTXUP33G53AWVYDPWTEPJ4A, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.017141162 (527h31m28.888224207s) [noise=true , address=145.79.227.7:4294 , piece_id=GOLFOMWUSPBLNQOVLHCNSV5TO7SYD6SXRQ4JH3Y6BMVODRFV376Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.030747159 (527h31m28.901830204s) [noise=true , address=145.79.227.41:4087 , piece_id=EZYX6T4M4HFZU2NQRXSKHDXHS7TCGNIO46JSULIVWESTUL4UYS6Q, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.257567985 (527h31m29.12865103s) [noise=true , address=45.91.116.11:4117 , piece_id=OUEJOCEHOAUFETJI6A6QBKJ3HBD3UPWD3DP5PM4BRFE2YY63NEIA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.281334640 (527h31m29.152417705s) [noise=true , address=77.90.31.11:4556 , piece_id=FNKGPJTHTOKTEZHIUXA6TT6HSP5CI2FZ5Q26KSFWVMKOFYSG2IXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.282698127 (527h31m29.153781172s) [noise=true , address=45.91.116.7:4009 , piece_id=6VJL67UXAFFTHELPOZRPBYSKCYW6ROXGNXLLPEO72OBRXEPYD7YQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.286111437 (527h31m29.157194482s) [noise=true , address=77.90.31.21:4878 , piece_id=K4XG57Q6NRCCEP2YQBKW6JHNGGJSZFPZ2NUKEWR4MX36D35NUHFQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.335636440 (527h31m29.206719475s) [noise=true , address=45.91.116.6:4974 , piece_id=2JQ2WJ2KGOSWXKFLZH3PIFCMTTSECVD5T6VO3GMUEOPCGWRHVIVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.398278622 (527h31m29.269361657s) [noise=true , address=45.91.116.10:4082 , piece_id=3FIUC3KWW5KIZSWJPT3AMNVDR2H4WF6DBQCJRWZU5KSJLXESKRVA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: <nil>
Dec 27 15:00:29.398299693 (527h31m29.269382738s) [seg_pos={0 0}, upload=462695]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:00:29.398467242 (527h31m29.269550287s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58496->77.90.31.15:4700: use of closed network connection
Dec 27 15:00:29.398473864 (527h31m29.269556899s) [noise=true , address=77.90.31.15:4700 , piece_id=KVP7WSIWFZIFXEMAFZ7QKA6ASQURNZNJMXUYLEU33OBWZJ3TECXA, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398508831 (527h31m29.269591886s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 27 15:00:29.398524260 (527h31m29.269607305s) [noise=true , address=45.91.116.9:4264 , piece_id=N62EQFPCQTHE7EJLGCAIHZO3XKKKI7LEIUFHXXH3NPRVXB5HM7IQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398683263 (527h31m29.269766308s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50870->77.90.31.12:4589: use of closed network connection
Dec 27 15:00:29.398689976 (527h31m29.269773021s) [noise=true , address=77.90.31.12:4589 , piece_id=DUXY6HIFSMCHAHULTTIAVBLGOYCOLGY6RFJP5OYTMHXFCWGJIWIQ, seg_pos={0 0}, upload=462695]: Long tail cancellation due to optimal threshold
Dec 27 15:00:29.398706236 (527h31m29.269789281s) [seg_pos={0 0}, upload=462695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:00:29.398771741 (527h31m29.269854776s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:00:29.398776370 (527h31m29.269859405s) [upload=462695]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:00:29.696725787 (527h31m29.567808842s) [upload=462695]: Done uploading segments...
Dec 27 15:00:29.696752318 (527h31m29.567835363s) [upload=462695]: Upload finished. err: <nil>
Duration: 30.003645656s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........X.............X..X...........X..............X....X.] duration:
49.543851284s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 15:00:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 49.543872564s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
8.053059672s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 15:00:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 8.053115569s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........X.............X..X...........X..............X....X] duration:
1m0.003275278s trace:
https://jaeger.dev.storj.io/trace/be41e8d6d19178d
FAIL (2025-12-27 14:59:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece...
Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment
Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments...
Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil>
Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection
Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF
Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold
Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments...
Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil>
Duration: 1m0.003299343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX...] duration:
3.443154058s trace:
https://jaeger.dev.storj.io/trace/42abe1eeecd8795d
FAIL (2025-12-27 14:59:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.443214082s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
14.995421756s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:59:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 14.995452134s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [........X..X....X.........X.X....X...X.....X..XX.......X..X.] duration:
5.721983291s trace:
https://jaeger.dev.storj.io/trace/e3bf4189c8fa585
FAIL (2025-12-27 14:59:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: YAQMAYGDF6U3ZN3DUEQGPFK4QG2X3RP2UCCZYL5DFU22X2E3UCCA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: GLFPQ27C4YLDZQ4JDPPVQDYGAWU7CZWMUA4DS4CDYOJ6KNY65RAA) context deadline exceeded
======LOGS======
Dec 27 14:58:36.130071013 (527h29m36.001154058s) [upload=462671]: Starting upload
Dec 27 14:58:36.130141678 (527h29m36.001224723s) [upload=462671]: Done starting upload
Dec 27 14:58:36.130229985 (527h29m36.00131303s) [upload=462671]: Uploading segments...
Dec 27 14:58:36.130336037 (527h29m36.001419082s) [upload=462671]: Got next segment. Inline: false
Dec 27 14:58:36.130383308 (527h29m36.001466353s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:58:36.130389219 (527h29m36.001472254s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:58:36.388319633 (527h29m36.259402678s) [seg_pos={0 0}, upload=462671]: Begin upload segment...
Dec 27 14:58:36.388374226 (527h29m36.259457261s) [seg_pos={0 0}, upload=462671]: Done begin upload segment.
Dec 27 14:58:36.388386740 (527h29m36.259469785s) [noise=true , address=145.79.227.44:4193 , piece_id=LHM7J6Y5SDO4KI7YJLAQFJ3MLNYP2ONNPQYJDBTUMJUZFCWVSPAQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.388401047 (527h29m36.259484082s) [noise=true , address=145.79.227.21:4474 , piece_id=MNTNBKFRMQLJEQKKJ4OGI5MJ4TGCHGQZNPHZ7GEGZKR4T6FIHQQA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.388889958 (527h29m36.259972993s) [noise=true , address=45.91.116.11:4120 , piece_id=GLFPQ27C4YLDZQ4JDPPVQDYGAWU7CZWMUA4DS4CDYOJ6KNY65RAA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389240716 (527h29m36.260323761s) [noise=true , address=145.79.227.4:4185 , piece_id=Q7HBPALNDQTP47AAHXJ3WTQUEOOWAYEXWD5AAS2Z4COXQE3UUAEQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389395652 (527h29m36.260478687s) [noise=true , address=145.79.227.3:4150 , piece_id=TXMGLQRQDX2BBZQQWITMKG7JUOVCC4WQIGJ4RSHXL5VKOCPV7TFQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389905462 (527h29m36.260988497s) [noise=true , address=45.91.116.10:4084 , piece_id=6SAM73BC4EDOZBMU3KX4OYHTEX447Y7PX2KLSIXE3SNMSBSTXP5Q, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389995514 (527h29m36.261078549s) [noise=true , address=45.91.116.8:4045 , piece_id=WAWK3KQRV4U2NHN7UX5DRPPBLNIBTVPMMEDOOZU45IAZGCMXGRMA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.390270868 (527h29m36.261353903s) [noise=true , address=145.79.227.48:4302 , piece_id=VOYL6NUYXVPGTBJ5EZHHGECCCEGV3QCCDOVBZMQJRGAZPBW2FC5Q, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.391015716 (527h29m36.262098761s) [noise=true , address=145.79.227.29:4942 , piece_id=FNJLT6K2H2T5VDBGPDTAYDQUAWPN74NGXR3RLUSVP5PT6E37LO5Q, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.392408618 (527h29m36.263491653s) [noise=true , address=45.91.116.14:4190 , piece_id=YAQMAYGDF6U3ZN3DUEQGPFK4QG2X3RP2UCCZYL5DFU22X2E3UCCA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.393394456 (527h29m36.264477501s) [noise=true , address=145.79.227.40:4049 , piece_id=GOOT4TCIDS2DKBXKRKWBE5SAL347CQJKBPHHWJCYGTZINZTK3HOA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.394473501 (527h29m36.265556536s) [noise=true , address=45.91.116.6:4975 , piece_id=FGHK73ABBNPXU44GNPLUNDGCOQ5NP6GHGQJEJN7BCJ6TDJWQOREQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.395627067 (527h29m36.266710112s) [noise=true , address=145.79.227.11:4732 , piece_id=A4LIKAAGBNVRWYYIOWUC4ZPML6O6FYCWMA5H3TYFAKTHISZIDCJA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.396692045 (527h29m36.2677751s) [noise=true , address=145.79.227.13:4802 , piece_id=7J7PAJSY2IJAHRNDILABDECPYMMQSDXUZONJSTBZ3IHSKPTKFNGA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.397048594 (527h29m36.268131629s) [noise=true , address=145.79.227.38:4978 , piece_id=Z3FDN2ASKOZDJFCE3S2LHCI725MVMPUDMQDI3Q5ZXAH6NKZP5ILQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.397622477 (527h29m36.268705512s) [noise=true , address=145.79.227.19:4078 , piece_id=POAW24SZXKSX723C35QHHFOZ7HB7YZK62XPMKVVEIW3KG7I3P3XQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:38.079112423 (527h29m37.950195458s) [upload=462671]: Next returned nil segment
Dec 27 14:58:38.079138152 (527h29m37.950221197s) [upload=462671]: Waiting for error group managing segments...
Dec 27 14:58:38.387983246 (527h29m38.259066281s) [noise=true , address=145.79.227.21:4474 , piece_id=MNTNBKFRMQLJEQKKJ4OGI5MJ4TGCHGQZNPHZ7GEGZKR4T6FIHQQA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.389939331 (527h29m38.261022376s) [noise=true , address=145.79.227.48:4302 , piece_id=VOYL6NUYXVPGTBJ5EZHHGECCCEGV3QCCDOVBZMQJRGAZPBW2FC5Q, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.399323811 (527h29m38.270406846s) [noise=true , address=145.79.227.4:4185 , piece_id=Q7HBPALNDQTP47AAHXJ3WTQUEOOWAYEXWD5AAS2Z4COXQE3UUAEQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.404673819 (527h29m38.275756864s) [noise=true , address=145.79.227.38:4978 , piece_id=Z3FDN2ASKOZDJFCE3S2LHCI725MVMPUDMQDI3Q5ZXAH6NKZP5ILQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.412823587 (527h29m38.283906632s) [noise=true , address=145.79.227.29:4942 , piece_id=FNJLT6K2H2T5VDBGPDTAYDQUAWPN74NGXR3RLUSVP5PT6E37LO5Q, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.423007939 (527h29m38.294090974s) [noise=true , address=145.79.227.13:4802 , piece_id=7J7PAJSY2IJAHRNDILABDECPYMMQSDXUZONJSTBZ3IHSKPTKFNGA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.423365832 (527h29m38.294448877s) [noise=true , address=145.79.227.3:4150 , piece_id=TXMGLQRQDX2BBZQQWITMKG7JUOVCC4WQIGJ4RSHXL5VKOCPV7TFQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.423695238 (527h29m38.294778273s) [noise=true , address=145.79.227.19:4078 , piece_id=POAW24SZXKSX723C35QHHFOZ7HB7YZK62XPMKVVEIW3KG7I3P3XQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.431503697 (527h29m38.302586732s) [noise=true , address=145.79.227.11:4732 , piece_id=A4LIKAAGBNVRWYYIOWUC4ZPML6O6FYCWMA5H3TYFAKTHISZIDCJA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.652246148 (527h29m38.523329193s) [noise=true , address=45.91.116.14:4190 , piece_id=YAQMAYGDF6U3ZN3DUEQGPFK4QG2X3RP2UCCZYL5DFU22X2E3UCCA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.652365496 (527h29m38.523448541s) [noise=true , address=45.91.116.11:4120 , piece_id=GLFPQ27C4YLDZQ4JDPPVQDYGAWU7CZWMUA4DS4CDYOJ6KNY65RAA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.655510324 (527h29m38.526593369s) [noise=true , address=45.91.116.6:4975 , piece_id=FGHK73ABBNPXU44GNPLUNDGCOQ5NP6GHGQJEJN7BCJ6TDJWQOREQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:39.096808181 (527h29m38.967891226s) [noise=true , address=45.91.116.10:4084 , piece_id=6SAM73BC4EDOZBMU3KX4OYHTEX447Y7PX2KLSIXE3SNMSBSTXP5Q, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:39.096832107 (527h29m38.967915142s) [seg_pos={0 0}, upload=462671]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:58:39.097053899 (527h29m38.968136944s) [noise=true , address=145.79.227.44:4193 , piece_id=LHM7J6Y5SDO4KI7YJLAQFJ3MLNYP2ONNPQYJDBTUMJUZFCWVSPAQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:58:39.097062134 (527h29m38.96814517s) [noise=true , address=145.79.227.44:4193 , piece_id=LHM7J6Y5SDO4KI7YJLAQFJ3MLNYP2ONNPQYJDBTUMJUZFCWVSPAQ, seg_pos={0 0}, upload=462671]: Long tail cancellation due to optimal threshold
Dec 27 14:58:39.097181923 (527h29m38.968264968s) [noise=true , address=45.91.116.8:4045 , piece_id=WAWK3KQRV4U2NHN7UX5DRPPBLNIBTVPMMEDOOZU45IAZGCMXGRMA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 27 14:58:39.097208593 (527h29m38.968291638s) [noise=true , address=45.91.116.8:4045 , piece_id=WAWK3KQRV4U2NHN7UX5DRPPBLNIBTVPMMEDOOZU45IAZGCMXGRMA, seg_pos={0 0}, upload=462671]: Long tail cancellation due to optimal threshold
Dec 27 14:58:39.097261645 (527h29m38.96834468s) [noise=true , address=145.79.227.40:4049 , piece_id=GOOT4TCIDS2DKBXKRKWBE5SAL347CQJKBPHHWJCYGTZINZTK3HOA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 27 14:58:39.097271884 (527h29m38.968354929s) [noise=true , address=145.79.227.40:4049 , piece_id=GOOT4TCIDS2DKBXKRKWBE5SAL347CQJKBPHHWJCYGTZINZTK3HOA, seg_pos={0 0}, upload=462671]: Long tail cancellation due to optimal threshold
Dec 27 14:58:39.097293165 (527h29m38.96837621s) [seg_pos={0 0}, upload=462671]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:58:39.097416499 (527h29m38.968499534s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:58:39.097432630 (527h29m38.968515705s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:58:39.390339024 (527h29m39.261422059s) [upload=462671]: Done uploading segments...
Dec 27 14:58:39.390355636 (527h29m39.261438671s) [upload=462671]: Upload finished. err: <nil>
Duration: 5.722010713s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [X.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX..] duration:
4.306448559s trace:
https://jaeger.dev.storj.io/trace/42abe1eeecd8795d
FAIL (2025-12-27 14:59:18 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.306477343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.........X..X....X.........X.X....X...X.....X..XX.......X..X] duration:
30.00317513s trace:
https://jaeger.dev.storj.io/trace/e3bf4189c8fa585
FAIL (2025-12-27 14:59:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: YAQMAYGDF6U3ZN3DUEQGPFK4QG2X3RP2UCCZYL5DFU22X2E3UCCA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: GLFPQ27C4YLDZQ4JDPPVQDYGAWU7CZWMUA4DS4CDYOJ6KNY65RAA) context deadline exceeded
======LOGS======
Dec 27 14:58:36.130071013 (527h29m36.001154058s) [upload=462671]: Starting upload
Dec 27 14:58:36.130141678 (527h29m36.001224723s) [upload=462671]: Done starting upload
Dec 27 14:58:36.130229985 (527h29m36.00131303s) [upload=462671]: Uploading segments...
Dec 27 14:58:36.130336037 (527h29m36.001419082s) [upload=462671]: Got next segment. Inline: false
Dec 27 14:58:36.130383308 (527h29m36.001466353s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:58:36.130389219 (527h29m36.001472254s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:58:36.388319633 (527h29m36.259402678s) [seg_pos={0 0}, upload=462671]: Begin upload segment...
Dec 27 14:58:36.388374226 (527h29m36.259457261s) [seg_pos={0 0}, upload=462671]: Done begin upload segment.
Dec 27 14:58:36.388386740 (527h29m36.259469785s) [noise=true , address=145.79.227.44:4193 , piece_id=LHM7J6Y5SDO4KI7YJLAQFJ3MLNYP2ONNPQYJDBTUMJUZFCWVSPAQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.388401047 (527h29m36.259484082s) [noise=true , address=145.79.227.21:4474 , piece_id=MNTNBKFRMQLJEQKKJ4OGI5MJ4TGCHGQZNPHZ7GEGZKR4T6FIHQQA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.388889958 (527h29m36.259972993s) [noise=true , address=45.91.116.11:4120 , piece_id=GLFPQ27C4YLDZQ4JDPPVQDYGAWU7CZWMUA4DS4CDYOJ6KNY65RAA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389240716 (527h29m36.260323761s) [noise=true , address=145.79.227.4:4185 , piece_id=Q7HBPALNDQTP47AAHXJ3WTQUEOOWAYEXWD5AAS2Z4COXQE3UUAEQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389395652 (527h29m36.260478687s) [noise=true , address=145.79.227.3:4150 , piece_id=TXMGLQRQDX2BBZQQWITMKG7JUOVCC4WQIGJ4RSHXL5VKOCPV7TFQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389905462 (527h29m36.260988497s) [noise=true , address=45.91.116.10:4084 , piece_id=6SAM73BC4EDOZBMU3KX4OYHTEX447Y7PX2KLSIXE3SNMSBSTXP5Q, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.389995514 (527h29m36.261078549s) [noise=true , address=45.91.116.8:4045 , piece_id=WAWK3KQRV4U2NHN7UX5DRPPBLNIBTVPMMEDOOZU45IAZGCMXGRMA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.390270868 (527h29m36.261353903s) [noise=true , address=145.79.227.48:4302 , piece_id=VOYL6NUYXVPGTBJ5EZHHGECCCEGV3QCCDOVBZMQJRGAZPBW2FC5Q, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.391015716 (527h29m36.262098761s) [noise=true , address=145.79.227.29:4942 , piece_id=FNJLT6K2H2T5VDBGPDTAYDQUAWPN74NGXR3RLUSVP5PT6E37LO5Q, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.392408618 (527h29m36.263491653s) [noise=true , address=45.91.116.14:4190 , piece_id=YAQMAYGDF6U3ZN3DUEQGPFK4QG2X3RP2UCCZYL5DFU22X2E3UCCA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.393394456 (527h29m36.264477501s) [noise=true , address=145.79.227.40:4049 , piece_id=GOOT4TCIDS2DKBXKRKWBE5SAL347CQJKBPHHWJCYGTZINZTK3HOA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.394473501 (527h29m36.265556536s) [noise=true , address=45.91.116.6:4975 , piece_id=FGHK73ABBNPXU44GNPLUNDGCOQ5NP6GHGQJEJN7BCJ6TDJWQOREQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.395627067 (527h29m36.266710112s) [noise=true , address=145.79.227.11:4732 , piece_id=A4LIKAAGBNVRWYYIOWUC4ZPML6O6FYCWMA5H3TYFAKTHISZIDCJA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.396692045 (527h29m36.2677751s) [noise=true , address=145.79.227.13:4802 , piece_id=7J7PAJSY2IJAHRNDILABDECPYMMQSDXUZONJSTBZ3IHSKPTKFNGA, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.397048594 (527h29m36.268131629s) [noise=true , address=145.79.227.38:4978 , piece_id=Z3FDN2ASKOZDJFCE3S2LHCI725MVMPUDMQDI3Q5ZXAH6NKZP5ILQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:36.397622477 (527h29m36.268705512s) [noise=true , address=145.79.227.19:4078 , piece_id=POAW24SZXKSX723C35QHHFOZ7HB7YZK62XPMKVVEIW3KG7I3P3XQ, seg_pos={0 0}, upload=462671]: Uploading piece...
Dec 27 14:58:38.079112423 (527h29m37.950195458s) [upload=462671]: Next returned nil segment
Dec 27 14:58:38.079138152 (527h29m37.950221197s) [upload=462671]: Waiting for error group managing segments...
Dec 27 14:58:38.387983246 (527h29m38.259066281s) [noise=true , address=145.79.227.21:4474 , piece_id=MNTNBKFRMQLJEQKKJ4OGI5MJ4TGCHGQZNPHZ7GEGZKR4T6FIHQQA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.389939331 (527h29m38.261022376s) [noise=true , address=145.79.227.48:4302 , piece_id=VOYL6NUYXVPGTBJ5EZHHGECCCEGV3QCCDOVBZMQJRGAZPBW2FC5Q, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.399323811 (527h29m38.270406846s) [noise=true , address=145.79.227.4:4185 , piece_id=Q7HBPALNDQTP47AAHXJ3WTQUEOOWAYEXWD5AAS2Z4COXQE3UUAEQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.404673819 (527h29m38.275756864s) [noise=true , address=145.79.227.38:4978 , piece_id=Z3FDN2ASKOZDJFCE3S2LHCI725MVMPUDMQDI3Q5ZXAH6NKZP5ILQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.412823587 (527h29m38.283906632s) [noise=true , address=145.79.227.29:4942 , piece_id=FNJLT6K2H2T5VDBGPDTAYDQUAWPN74NGXR3RLUSVP5PT6E37LO5Q, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.423007939 (527h29m38.294090974s) [noise=true , address=145.79.227.13:4802 , piece_id=7J7PAJSY2IJAHRNDILABDECPYMMQSDXUZONJSTBZ3IHSKPTKFNGA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.423365832 (527h29m38.294448877s) [noise=true , address=145.79.227.3:4150 , piece_id=TXMGLQRQDX2BBZQQWITMKG7JUOVCC4WQIGJ4RSHXL5VKOCPV7TFQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.423695238 (527h29m38.294778273s) [noise=true , address=145.79.227.19:4078 , piece_id=POAW24SZXKSX723C35QHHFOZ7HB7YZK62XPMKVVEIW3KG7I3P3XQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.431503697 (527h29m38.302586732s) [noise=true , address=145.79.227.11:4732 , piece_id=A4LIKAAGBNVRWYYIOWUC4ZPML6O6FYCWMA5H3TYFAKTHISZIDCJA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.652246148 (527h29m38.523329193s) [noise=true , address=45.91.116.14:4190 , piece_id=YAQMAYGDF6U3ZN3DUEQGPFK4QG2X3RP2UCCZYL5DFU22X2E3UCCA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.652365496 (527h29m38.523448541s) [noise=true , address=45.91.116.11:4120 , piece_id=GLFPQ27C4YLDZQ4JDPPVQDYGAWU7CZWMUA4DS4CDYOJ6KNY65RAA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:38.655510324 (527h29m38.526593369s) [noise=true , address=45.91.116.6:4975 , piece_id=FGHK73ABBNPXU44GNPLUNDGCOQ5NP6GHGQJEJN7BCJ6TDJWQOREQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:39.096808181 (527h29m38.967891226s) [noise=true , address=45.91.116.10:4084 , piece_id=6SAM73BC4EDOZBMU3KX4OYHTEX447Y7PX2KLSIXE3SNMSBSTXP5Q, seg_pos={0 0}, upload=462671]: Done uploading piece. err: <nil>
Dec 27 14:58:39.096832107 (527h29m38.967915142s) [seg_pos={0 0}, upload=462671]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:58:39.097053899 (527h29m38.968136944s) [noise=true , address=145.79.227.44:4193 , piece_id=LHM7J6Y5SDO4KI7YJLAQFJ3MLNYP2ONNPQYJDBTUMJUZFCWVSPAQ, seg_pos={0 0}, upload=462671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:58:39.097062134 (527h29m38.96814517s) [noise=true , address=145.79.227.44:4193 , piece_id=LHM7J6Y5SDO4KI7YJLAQFJ3MLNYP2ONNPQYJDBTUMJUZFCWVSPAQ, seg_pos={0 0}, upload=462671]: Long tail cancellation due to optimal threshold
Dec 27 14:58:39.097181923 (527h29m38.968264968s) [noise=true , address=45.91.116.8:4045 , piece_id=WAWK3KQRV4U2NHN7UX5DRPPBLNIBTVPMMEDOOZU45IAZGCMXGRMA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 27 14:58:39.097208593 (527h29m38.968291638s) [noise=true , address=45.91.116.8:4045 , piece_id=WAWK3KQRV4U2NHN7UX5DRPPBLNIBTVPMMEDOOZU45IAZGCMXGRMA, seg_pos={0 0}, upload=462671]: Long tail cancellation due to optimal threshold
Dec 27 14:58:39.097261645 (527h29m38.96834468s) [noise=true , address=145.79.227.40:4049 , piece_id=GOOT4TCIDS2DKBXKRKWBE5SAL347CQJKBPHHWJCYGTZINZTK3HOA, seg_pos={0 0}, upload=462671]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 27 14:58:39.097271884 (527h29m38.968354929s) [noise=true , address=145.79.227.40:4049 , piece_id=GOOT4TCIDS2DKBXKRKWBE5SAL347CQJKBPHHWJCYGTZINZTK3HOA, seg_pos={0 0}, upload=462671]: Long tail cancellation due to optimal threshold
Dec 27 14:58:39.097293165 (527h29m38.96837621s) [seg_pos={0 0}, upload=462671]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:58:39.097416499 (527h29m38.968499534s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:58:39.097432630 (527h29m38.968515705s) [upload=462671]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:58:39.390339024 (527h29m39.261422059s) [upload=462671]: Done uploading segments...
Dec 27 14:58:39.390355636 (527h29m39.261438671s) [upload=462671]: Upload finished. err: <nil>
Duration: 30.003191831s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
7.677039646s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:58:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 7.677065104s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X.............X..X...........X..............X....] duration:
40.210232035s trace:
https://jaeger.dev.storj.io/trace/167c0ca1efdcf531
FAIL (2025-12-27 14:58:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3]: Uploading piece...
Dec 27 14:53:39.160172442 (527h24m39.031255487s) [noise=true , address=77.90.31.17:4792 , piece_id=E4XYPNEP33XARAQECIXL3JZ36DKUOIWL23LXS73WVBHBM3BVDNEQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160453677 (527h24m39.031536722s) [noise=true , address=45.91.116.4:4964 , piece_id=6LYWNDSTB4QDVPY24SJT562RSPMYMN5BYME3BTLYJVUSTTJ5R7IQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160608532 (527h24m39.031691567s) [noise=true , address=77.90.31.14:4671 , piece_id=3USEZRVYLAUQV7GCHSE3VOTIVGGLQXRMIHW54YLDQ6NIIYXWTZIQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160718932 (527h24m39.031801967s) [noise=true , address=77.90.31.16:4744 , piece_id=PTF2RZGMBQ3QISOUEFK26WLH6GJIEHBU2SCJVJ3FTN6YCT7VITVQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160990800 (527h24m39.032073845s) [noise=true , address=64.49.13.169:4715 , piece_id=RZQ5LW6ZUUN72RAOJM6X7VJGVLR2PF6KUPQOPFM52UCY22IYJRJQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161023963 (527h24m39.032107008s) [noise=true , address=45.91.116.9:4286 , piece_id=DASZFAA3EV6G5DJ5WK7LNZKJ2AOTOLE2L7VM7SI2VRR6OM46ABYA, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161258049 (527h24m39.032341094s) [noise=true , address=77.90.31.15:4731 , piece_id=RJO5ZU4PTU3LX3GHMASZH7XAC4E45O3IEIJ3B6RLHAEBGRBUOBEQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161403075 (527h24m39.03248612s) [noise=true , address=145.79.227.40:4084 , piece_id=3IFOK6IYHB7A5UZNXFJTSPM64TMFEFV7X4XES7WQHUCBHBJ7F3OA, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161570003 (527h24m39.032653038s) [noise=true , address=77.90.31.21:4897 , piece_id=TPWCDIXFASV53CQX6GEXVXZUWBOTXEXXHXSEKYFVVOOQQARALTQA, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161870705 (527h24m39.03295375s) [noise=true , address=145.79.227.30:4787 , piece_id=VPQC3Y7NV2RM4VDEVHS5LHC2SITTSDUATGNLHAYIY5X56A22UBQQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.162069173 (527h24m39.033152258s) [noise=true , address=77.90.31.10:4521 , piece_id=QH5LGSQ43VA2EWTOEI24CACPZNWIZLQTS7LHTX427J7JMLL22RYQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.162305944 (527h24m39.033388979s) [noise=true , address=77.90.31.18:4826 , piece_id=GDJWD7TXJYDJBGSKBKKDPVGXRWX4YV6X3BRDH7RLMYH7MT5KAEEQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.162696397 (527h24m39.033779442s) [noise=true , address=45.91.116.6:4999 , piece_id=K7UP5EPAF3SFRKRLDUIFG3JCZP4AYGTAD5WZLKKELTEQQYR3ZZAQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.163075309 (527h24m39.034158354s) [noise=true , address=145.79.227.34:4912 , piece_id=2DRRQZNEGMYV6QCTZL6YFCEKLBARAKRCAN6RLDW5ISF47XZYYYUQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:40.469387573 (527h24m40.340470658s) [upload=462593]: Next returned nil segment
Dec 27 14:53:40.469411738 (527h24m40.340494833s) [upload=462593]: Waiting for error group managing segments...
Dec 27 14:53:40.818341341 (527h24m40.689424376s) [noise=true , address=64.49.13.140:4505 , piece_id=OFDHGZFYCZSKPBYPBBGWODDD7NONMP2OSUDM3TNCKZJX7XSDYQZQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.211841824 (527h24m41.082924859s) [noise=true , address=77.90.31.10:4521 , piece_id=QH5LGSQ43VA2EWTOEI24CACPZNWIZLQTS7LHTX427J7JMLL22RYQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.212248829 (527h24m41.083331864s) [noise=true , address=77.90.31.13:4625 , piece_id=AFKIH7L7SX6APDQE4Z43GB4ORQHIUY4AMFVMRSECZQZCRBRIJTAA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.228401325 (527h24m41.09948436s) [noise=true , address=77.90.31.19:4846 , piece_id=DWUMLSPTP6UP3MRTPGKVPLWGF7MANOM7QQ5RIXWZGKBOZMTLPWNA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.249636571 (527h24m41.120719616s) [noise=true , address=77.90.31.17:4792 , piece_id=E4XYPNEP33XARAQECIXL3JZ36DKUOIWL23LXS73WVBHBM3BVDNEQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.299817324 (527h24m41.170900359s) [noise=true , address=77.90.31.16:4744 , piece_id=PTF2RZGMBQ3QISOUEFK26WLH6GJIEHBU2SCJVJ3FTN6YCT7VITVQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.325550715 (527h24m41.19663376s) [noise=true , address=77.90.31.18:4826 , piece_id=GDJWD7TXJYDJBGSKBKKDPVGXRWX4YV6X3BRDH7RLMYH7MT5KAEEQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.487623620 (527h24m41.358706665s) [noise=true , address=45.91.116.4:4964 , piece_id=6LYWNDSTB4QDVPY24SJT562RSPMYMN5BYME3BTLYJVUSTTJ5R7IQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.546692006 (527h24m41.417775041s) [noise=true , address=77.90.31.14:4671 , piece_id=3USEZRVYLAUQV7GCHSE3VOTIVGGLQXRMIHW54YLDQ6NIIYXWTZIQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.013323652 (527h24m41.884406698s) [noise=true , address=145.79.227.40:4084 , piece_id=3IFOK6IYHB7A5UZNXFJTSPM64TMFEFV7X4XES7WQHUCBHBJ7F3OA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.044636839 (527h24m41.915719884s) [noise=true , address=145.79.227.30:4787 , piece_id=VPQC3Y7NV2RM4VDEVHS5LHC2SITTSDUATGNLHAYIY5X56A22UBQQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.128054029 (527h24m41.999137094s) [noise=true , address=145.79.227.34:4912 , piece_id=2DRRQZNEGMYV6QCTZL6YFCEKLBARAKRCAN6RLDW5ISF47XZYYYUQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.317704146 (527h24m42.188787211s) [noise=true , address=45.91.116.9:4286 , piece_id=DASZFAA3EV6G5DJ5WK7LNZKJ2AOTOLE2L7VM7SI2VRR6OM46ABYA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.476286533 (527h24m42.347369568s) [noise=true , address=45.91.116.6:4999 , piece_id=K7UP5EPAF3SFRKRLDUIFG3JCZP4AYGTAD5WZLKKELTEQQYR3ZZAQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.476308165 (527h24m42.34739121s) [seg_pos={0 0}, upload=462593]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:53:42.476480021 (527h24m42.347563066s) [noise=true , address=64.49.13.169:4715 , piece_id=RZQ5LW6ZUUN72RAOJM6X7VJGVLR2PF6KUPQOPFM52UCY22IYJRJQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52060->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52060->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52060->64.49.13.169:4715: use of closed network connection
Dec 27 14:53:42.476486284 (527h24m42.347569319s) [noise=true , address=64.49.13.169:4715 , piece_id=RZQ5LW6ZUUN72RAOJM6X7VJGVLR2PF6KUPQOPFM52UCY22IYJRJQ, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476607795 (527h24m42.34769083s) [noise=true , address=45.91.116.11:4124 , piece_id=KU3SW2A3FOFMNQ44KRZSZKWUYHYLXJEVLKIZS5JCKEKY3J3FVEXA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vSQgozKbq7XjSYP5dUJjkrnTkWX4EFgBkruqUmdV3ehnoPfkr): protocol: expected piece hash; context canceled; EOF
Dec 27 14:53:42.476613045 (527h24m42.34769608s) [noise=true , address=45.91.116.11:4124 , piece_id=KU3SW2A3FOFMNQ44KRZSZKWUYHYLXJEVLKIZS5JCKEKY3J3FVEXA, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476689942 (527h24m42.347772977s) [noise=true , address=77.90.31.21:4897 , piece_id=TPWCDIXFASV53CQX6GEXVXZUWBOTXEXXHXSEKYFVVOOQQARALTQA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:55336->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:55336->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55336->77.90.31.21:4897: use of closed network connection
Dec 27 14:53:42.476694791 (527h24m42.347777826s) [noise=true , address=77.90.31.21:4897 , piece_id=TPWCDIXFASV53CQX6GEXVXZUWBOTXEXXHXSEKYFVVOOQQARALTQA, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476797817 (527h24m42.347880872s) [noise=true , address=77.90.31.15:4731 , piece_id=RJO5ZU4PTU3LX3GHMASZH7XAC4E45O3IEIJ3B6RLHAEBGRBUOBEQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: read tcp 188.166.225.170:54992->77.90.31.15:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 14:53:42.476815029 (527h24m42.347898085s) [noise=true , address=77.90.31.15:4731 , piece_id=RJO5ZU4PTU3LX3GHMASZH7XAC4E45O3IEIJ3B6RLHAEBGRBUOBEQ, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476861518 (527h24m42.347944563s) [seg_pos={0 0}, upload=462593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:53:42.476991285 (527h24m42.34807433s) [upload=462593]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:53:42.476997887 (527h24m42.348080943s) [upload=462593]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:53:42.771143972 (527h24m42.642227018s) [upload=462593]: Done uploading segments...
Dec 27 14:53:42.771178699 (527h24m42.642261744s) [upload=462593]: Upload finished. err: <nil>
Duration: 40.210261439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.] duration:
5.001508953s trace:
https://jaeger.dev.storj.io/trace/42abe1eeecd8795d
FAIL (2025-12-27 14:58:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001542437s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
7.672699592s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:58:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 7.672720882s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX] duration:
5.001337408s trace:
https://jaeger.dev.storj.io/trace/42abe1eeecd8795d
FAIL (2025-12-27 14:57:51 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001361695s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X.............X..X...........X..............X...] duration:
53.393530375s trace:
https://jaeger.dev.storj.io/trace/167c0ca1efdcf531
FAIL (2025-12-27 14:57:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3]: Uploading piece...
Dec 27 14:53:39.160172442 (527h24m39.031255487s) [noise=true , address=77.90.31.17:4792 , piece_id=E4XYPNEP33XARAQECIXL3JZ36DKUOIWL23LXS73WVBHBM3BVDNEQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160453677 (527h24m39.031536722s) [noise=true , address=45.91.116.4:4964 , piece_id=6LYWNDSTB4QDVPY24SJT562RSPMYMN5BYME3BTLYJVUSTTJ5R7IQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160608532 (527h24m39.031691567s) [noise=true , address=77.90.31.14:4671 , piece_id=3USEZRVYLAUQV7GCHSE3VOTIVGGLQXRMIHW54YLDQ6NIIYXWTZIQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160718932 (527h24m39.031801967s) [noise=true , address=77.90.31.16:4744 , piece_id=PTF2RZGMBQ3QISOUEFK26WLH6GJIEHBU2SCJVJ3FTN6YCT7VITVQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.160990800 (527h24m39.032073845s) [noise=true , address=64.49.13.169:4715 , piece_id=RZQ5LW6ZUUN72RAOJM6X7VJGVLR2PF6KUPQOPFM52UCY22IYJRJQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161023963 (527h24m39.032107008s) [noise=true , address=45.91.116.9:4286 , piece_id=DASZFAA3EV6G5DJ5WK7LNZKJ2AOTOLE2L7VM7SI2VRR6OM46ABYA, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161258049 (527h24m39.032341094s) [noise=true , address=77.90.31.15:4731 , piece_id=RJO5ZU4PTU3LX3GHMASZH7XAC4E45O3IEIJ3B6RLHAEBGRBUOBEQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161403075 (527h24m39.03248612s) [noise=true , address=145.79.227.40:4084 , piece_id=3IFOK6IYHB7A5UZNXFJTSPM64TMFEFV7X4XES7WQHUCBHBJ7F3OA, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161570003 (527h24m39.032653038s) [noise=true , address=77.90.31.21:4897 , piece_id=TPWCDIXFASV53CQX6GEXVXZUWBOTXEXXHXSEKYFVVOOQQARALTQA, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.161870705 (527h24m39.03295375s) [noise=true , address=145.79.227.30:4787 , piece_id=VPQC3Y7NV2RM4VDEVHS5LHC2SITTSDUATGNLHAYIY5X56A22UBQQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.162069173 (527h24m39.033152258s) [noise=true , address=77.90.31.10:4521 , piece_id=QH5LGSQ43VA2EWTOEI24CACPZNWIZLQTS7LHTX427J7JMLL22RYQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.162305944 (527h24m39.033388979s) [noise=true , address=77.90.31.18:4826 , piece_id=GDJWD7TXJYDJBGSKBKKDPVGXRWX4YV6X3BRDH7RLMYH7MT5KAEEQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.162696397 (527h24m39.033779442s) [noise=true , address=45.91.116.6:4999 , piece_id=K7UP5EPAF3SFRKRLDUIFG3JCZP4AYGTAD5WZLKKELTEQQYR3ZZAQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:39.163075309 (527h24m39.034158354s) [noise=true , address=145.79.227.34:4912 , piece_id=2DRRQZNEGMYV6QCTZL6YFCEKLBARAKRCAN6RLDW5ISF47XZYYYUQ, seg_pos={0 0}, upload=462593]: Uploading piece...
Dec 27 14:53:40.469387573 (527h24m40.340470658s) [upload=462593]: Next returned nil segment
Dec 27 14:53:40.469411738 (527h24m40.340494833s) [upload=462593]: Waiting for error group managing segments...
Dec 27 14:53:40.818341341 (527h24m40.689424376s) [noise=true , address=64.49.13.140:4505 , piece_id=OFDHGZFYCZSKPBYPBBGWODDD7NONMP2OSUDM3TNCKZJX7XSDYQZQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.211841824 (527h24m41.082924859s) [noise=true , address=77.90.31.10:4521 , piece_id=QH5LGSQ43VA2EWTOEI24CACPZNWIZLQTS7LHTX427J7JMLL22RYQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.212248829 (527h24m41.083331864s) [noise=true , address=77.90.31.13:4625 , piece_id=AFKIH7L7SX6APDQE4Z43GB4ORQHIUY4AMFVMRSECZQZCRBRIJTAA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.228401325 (527h24m41.09948436s) [noise=true , address=77.90.31.19:4846 , piece_id=DWUMLSPTP6UP3MRTPGKVPLWGF7MANOM7QQ5RIXWZGKBOZMTLPWNA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.249636571 (527h24m41.120719616s) [noise=true , address=77.90.31.17:4792 , piece_id=E4XYPNEP33XARAQECIXL3JZ36DKUOIWL23LXS73WVBHBM3BVDNEQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.299817324 (527h24m41.170900359s) [noise=true , address=77.90.31.16:4744 , piece_id=PTF2RZGMBQ3QISOUEFK26WLH6GJIEHBU2SCJVJ3FTN6YCT7VITVQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.325550715 (527h24m41.19663376s) [noise=true , address=77.90.31.18:4826 , piece_id=GDJWD7TXJYDJBGSKBKKDPVGXRWX4YV6X3BRDH7RLMYH7MT5KAEEQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.487623620 (527h24m41.358706665s) [noise=true , address=45.91.116.4:4964 , piece_id=6LYWNDSTB4QDVPY24SJT562RSPMYMN5BYME3BTLYJVUSTTJ5R7IQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:41.546692006 (527h24m41.417775041s) [noise=true , address=77.90.31.14:4671 , piece_id=3USEZRVYLAUQV7GCHSE3VOTIVGGLQXRMIHW54YLDQ6NIIYXWTZIQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.013323652 (527h24m41.884406698s) [noise=true , address=145.79.227.40:4084 , piece_id=3IFOK6IYHB7A5UZNXFJTSPM64TMFEFV7X4XES7WQHUCBHBJ7F3OA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.044636839 (527h24m41.915719884s) [noise=true , address=145.79.227.30:4787 , piece_id=VPQC3Y7NV2RM4VDEVHS5LHC2SITTSDUATGNLHAYIY5X56A22UBQQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.128054029 (527h24m41.999137094s) [noise=true , address=145.79.227.34:4912 , piece_id=2DRRQZNEGMYV6QCTZL6YFCEKLBARAKRCAN6RLDW5ISF47XZYYYUQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.317704146 (527h24m42.188787211s) [noise=true , address=45.91.116.9:4286 , piece_id=DASZFAA3EV6G5DJ5WK7LNZKJ2AOTOLE2L7VM7SI2VRR6OM46ABYA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.476286533 (527h24m42.347369568s) [noise=true , address=45.91.116.6:4999 , piece_id=K7UP5EPAF3SFRKRLDUIFG3JCZP4AYGTAD5WZLKKELTEQQYR3ZZAQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: <nil>
Dec 27 14:53:42.476308165 (527h24m42.34739121s) [seg_pos={0 0}, upload=462593]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:53:42.476480021 (527h24m42.347563066s) [noise=true , address=64.49.13.169:4715 , piece_id=RZQ5LW6ZUUN72RAOJM6X7VJGVLR2PF6KUPQOPFM52UCY22IYJRJQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:52060->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:52060->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52060->64.49.13.169:4715: use of closed network connection
Dec 27 14:53:42.476486284 (527h24m42.347569319s) [noise=true , address=64.49.13.169:4715 , piece_id=RZQ5LW6ZUUN72RAOJM6X7VJGVLR2PF6KUPQOPFM52UCY22IYJRJQ, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476607795 (527h24m42.34769083s) [noise=true , address=45.91.116.11:4124 , piece_id=KU3SW2A3FOFMNQ44KRZSZKWUYHYLXJEVLKIZS5JCKEKY3J3FVEXA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vSQgozKbq7XjSYP5dUJjkrnTkWX4EFgBkruqUmdV3ehnoPfkr): protocol: expected piece hash; context canceled; EOF
Dec 27 14:53:42.476613045 (527h24m42.34769608s) [noise=true , address=45.91.116.11:4124 , piece_id=KU3SW2A3FOFMNQ44KRZSZKWUYHYLXJEVLKIZS5JCKEKY3J3FVEXA, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476689942 (527h24m42.347772977s) [noise=true , address=77.90.31.21:4897 , piece_id=TPWCDIXFASV53CQX6GEXVXZUWBOTXEXXHXSEKYFVVOOQQARALTQA, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:55336->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:55336->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55336->77.90.31.21:4897: use of closed network connection
Dec 27 14:53:42.476694791 (527h24m42.347777826s) [noise=true , address=77.90.31.21:4897 , piece_id=TPWCDIXFASV53CQX6GEXVXZUWBOTXEXXHXSEKYFVVOOQQARALTQA, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476797817 (527h24m42.347880872s) [noise=true , address=77.90.31.15:4731 , piece_id=RJO5ZU4PTU3LX3GHMASZH7XAC4E45O3IEIJ3B6RLHAEBGRBUOBEQ, seg_pos={0 0}, upload=462593]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: read tcp 188.166.225.170:54992->77.90.31.15:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 14:53:42.476815029 (527h24m42.347898085s) [noise=true , address=77.90.31.15:4731 , piece_id=RJO5ZU4PTU3LX3GHMASZH7XAC4E45O3IEIJ3B6RLHAEBGRBUOBEQ, seg_pos={0 0}, upload=462593]: Long tail cancellation due to optimal threshold
Dec 27 14:53:42.476861518 (527h24m42.347944563s) [seg_pos={0 0}, upload=462593]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:53:42.476991285 (527h24m42.34807433s) [upload=462593]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:53:42.476997887 (527h24m42.348080943s) [upload=462593]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:53:42.771143972 (527h24m42.642227018s) [upload=462593]: Done uploading segments...
Dec 27 14:53:42.771178699 (527h24m42.642261744s) [upload=462593]: Upload finished. err: <nil>
Duration: 53.393553548s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..........X..X....X.........X.X....X...X.....X..XX.......X..] duration:
5.976286677s trace:
https://jaeger.dev.storj.io/trace/2a63206377bc416d
FAIL (2025-12-27 14:57:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q) context deadline exceeded
======LOGS======
Dec 27 14:56:30.727029380 (527h27m30.598112415s) [upload=462637]: Starting upload
Dec 27 14:56:30.727138748 (527h27m30.598221823s) [upload=462637]: Done starting upload
Dec 27 14:56:30.727231635 (527h27m30.59831468s) [upload=462637]: Uploading segments...
Dec 27 14:56:30.727481190 (527h27m30.598564225s) [upload=462637]: Got next segment. Inline: false
Dec 27 14:56:30.727521727 (527h27m30.598604782s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:56:30.727527999 (527h27m30.598611054s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:56:30.978256560 (527h27m30.849339605s) [seg_pos={0 0}, upload=462637]: Begin upload segment...
Dec 27 14:56:30.978308969 (527h27m30.849392004s) [seg_pos={0 0}, upload=462637]: Done begin upload segment.
Dec 27 14:56:30.978341601 (527h27m30.849424646s) [noise=true , address=145.79.227.45:4231 , piece_id=YDXPUTXI7RDN5FTA7IZOGFBDOLKGFLFCXDOTCQITRCU7NAFCMW5Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.978358374 (527h27m30.849441419s) [noise=true , address=145.79.227.11:4730 , piece_id=66IT4MEAZJW6HPXGD4AZY2IXI47VZVJYVTHILSYA4CJPWTUWFK5A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.978981050 (527h27m30.850064085s) [noise=true , address=145.79.227.32:4834 , piece_id=E2WAHKFHP2BIBWSMR5UZ6YEOI3EVVS6LBTGBUMSXIGSGXW3B5KWQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979504376 (527h27m30.850587411s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979586001 (527h27m30.850669036s) [noise=true , address=145.79.227.5:4221 , piece_id=CWIDO7AWOOQZZBPBCDRIJTW3BHJ4ZEJKOFT3XUB6MXEN3NGUAUNQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979878088 (527h27m30.850961123s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979975422 (527h27m30.851058498s) [noise=true , address=77.90.31.18:4805 , piece_id=VTKOIICB6NXC5O7RY24NNFZ3MHTC7IRUXI7XNTKOPTN65PDXBRUQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980238263 (527h27m30.851321298s) [noise=true , address=45.91.116.4:4937 , piece_id=MPZ6KR5W5SI2V4L6ERAPSKV6OW4R3HY4BKJ6O3JNDVNJRZXFFN7A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980583411 (527h27m30.851666446s) [noise=true , address=77.90.31.10:4517 , piece_id=6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980937996 (527h27m30.852021041s) [noise=true , address=45.91.116.11:4120 , piece_id=GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981235923 (527h27m30.852318959s) [noise=true , address=145.79.227.7:4295 , piece_id=SRIH4SPOG6ANG4DIYUHM2XAVDDYLPPNE3ANMJI6MEEZJKGQAB56A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981510996 (527h27m30.852594031s) [noise=true , address=145.79.227.49:4338 , piece_id=K22DIQETHGNUQP56JYROWAJW5ZSRHZY4SBHFJ7ZY75ZQXRCOUTNQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981853198 (527h27m30.852936233s) [noise=true , address=145.79.227.51:4410 , piece_id=35UO43XMARP5UYE2R7W4VZY72P76WIBKFGXXVW3NW7SBW44JOK2Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.982462639 (527h27m30.853545694s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.982947583 (527h27m30.854030628s) [noise=true , address=145.79.227.52:4447 , piece_id=XKDSTWXNIZ4L2V4TKCRSMDAKL2KEK5FC5WCNWYEGYHA7MD3SYWGA, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.983327466 (527h27m30.854410501s) [noise=true , address=45.91.116.9:4262 , piece_id=KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:32.605493069 (527h27m32.476576114s) [upload=462637]: Next returned nil segment
Dec 27 14:56:32.605509640 (527h27m32.476592685s) [upload=462637]: Waiting for error group managing segments...
Dec 27 14:56:32.989144216 (527h27m32.860227261s) [noise=true , address=145.79.227.51:4410 , piece_id=35UO43XMARP5UYE2R7W4VZY72P76WIBKFGXXVW3NW7SBW44JOK2Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.990269098 (527h27m32.861352154s) [noise=true , address=145.79.227.52:4447 , piece_id=XKDSTWXNIZ4L2V4TKCRSMDAKL2KEK5FC5WCNWYEGYHA7MD3SYWGA, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.990408243 (527h27m32.861491278s) [noise=true , address=145.79.227.32:4834 , piece_id=E2WAHKFHP2BIBWSMR5UZ6YEOI3EVVS6LBTGBUMSXIGSGXW3B5KWQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.994972074 (527h27m32.866055109s) [noise=true , address=145.79.227.49:4338 , piece_id=K22DIQETHGNUQP56JYROWAJW5ZSRHZY4SBHFJ7ZY75ZQXRCOUTNQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.005702276 (527h27m32.876785341s) [noise=true , address=145.79.227.5:4221 , piece_id=CWIDO7AWOOQZZBPBCDRIJTW3BHJ4ZEJKOFT3XUB6MXEN3NGUAUNQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.007088375 (527h27m32.87817142s) [noise=true , address=145.79.227.11:4730 , piece_id=66IT4MEAZJW6HPXGD4AZY2IXI47VZVJYVTHILSYA4CJPWTUWFK5A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.010248613 (527h27m32.881331648s) [noise=true , address=145.79.227.7:4295 , piece_id=SRIH4SPOG6ANG4DIYUHM2XAVDDYLPPNE3ANMJI6MEEZJKGQAB56A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.015750199 (527h27m32.886833244s) [noise=true , address=145.79.227.45:4231 , piece_id=YDXPUTXI7RDN5FTA7IZOGFBDOLKGFLFCXDOTCQITRCU7NAFCMW5Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.253649410 (527h27m33.124732455s) [noise=true , address=45.91.116.9:4262 , piece_id=KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.269921144 (527h27m33.141004189s) [noise=true , address=45.91.116.11:4120 , piece_id=GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.282601289 (527h27m33.153684334s) [noise=true , address=77.90.31.18:4805 , piece_id=VTKOIICB6NXC5O7RY24NNFZ3MHTC7IRUXI7XNTKOPTN65PDXBRUQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.286531783 (527h27m33.157614818s) [noise=true , address=45.91.116.4:4937 , piece_id=MPZ6KR5W5SI2V4L6ERAPSKV6OW4R3HY4BKJ6O3JNDVNJRZXFFN7A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.299383785 (527h27m33.17046683s) [noise=true , address=77.90.31.10:4517 , piece_id=6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.299406449 (527h27m33.170489494s) [seg_pos={0 0}, upload=462637]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:56:33.299766815 (527h27m33.17084987s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.299776964 (527h27m33.170860009s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300183097 (527h27m33.171266152s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.300192395 (527h27m33.17127543s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300370494 (527h27m33.171453539s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.300436740 (527h27m33.171519775s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300461508 (527h27m33.171544553s) [seg_pos={0 0}, upload=462637]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:56:33.300544585 (527h27m33.17162763s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:56:33.300550718 (527h27m33.171633763s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:56:33.591506157 (527h27m33.462589202s) [upload=462637]: Done uploading segments...
Dec 27 14:56:33.591529251 (527h27m33.462612286s) [upload=462637]: Upload finished. err: <nil>
Duration: 5.976314189s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
10.296429143s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:57:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 10.296466785s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXX] duration:
5.001914856s trace:
https://jaeger.dev.storj.io/trace/52e3344e3e690efd
FAIL (2025-12-27 14:57:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001938902s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...........X..X....X.........X.X....X...X.....X..XX.......X.] duration:
5.663700541s trace:
https://jaeger.dev.storj.io/trace/2a63206377bc416d
FAIL (2025-12-27 14:57:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q) context deadline exceeded
======LOGS======
Dec 27 14:56:30.727029380 (527h27m30.598112415s) [upload=462637]: Starting upload
Dec 27 14:56:30.727138748 (527h27m30.598221823s) [upload=462637]: Done starting upload
Dec 27 14:56:30.727231635 (527h27m30.59831468s) [upload=462637]: Uploading segments...
Dec 27 14:56:30.727481190 (527h27m30.598564225s) [upload=462637]: Got next segment. Inline: false
Dec 27 14:56:30.727521727 (527h27m30.598604782s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:56:30.727527999 (527h27m30.598611054s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:56:30.978256560 (527h27m30.849339605s) [seg_pos={0 0}, upload=462637]: Begin upload segment...
Dec 27 14:56:30.978308969 (527h27m30.849392004s) [seg_pos={0 0}, upload=462637]: Done begin upload segment.
Dec 27 14:56:30.978341601 (527h27m30.849424646s) [noise=true , address=145.79.227.45:4231 , piece_id=YDXPUTXI7RDN5FTA7IZOGFBDOLKGFLFCXDOTCQITRCU7NAFCMW5Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.978358374 (527h27m30.849441419s) [noise=true , address=145.79.227.11:4730 , piece_id=66IT4MEAZJW6HPXGD4AZY2IXI47VZVJYVTHILSYA4CJPWTUWFK5A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.978981050 (527h27m30.850064085s) [noise=true , address=145.79.227.32:4834 , piece_id=E2WAHKFHP2BIBWSMR5UZ6YEOI3EVVS6LBTGBUMSXIGSGXW3B5KWQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979504376 (527h27m30.850587411s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979586001 (527h27m30.850669036s) [noise=true , address=145.79.227.5:4221 , piece_id=CWIDO7AWOOQZZBPBCDRIJTW3BHJ4ZEJKOFT3XUB6MXEN3NGUAUNQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979878088 (527h27m30.850961123s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979975422 (527h27m30.851058498s) [noise=true , address=77.90.31.18:4805 , piece_id=VTKOIICB6NXC5O7RY24NNFZ3MHTC7IRUXI7XNTKOPTN65PDXBRUQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980238263 (527h27m30.851321298s) [noise=true , address=45.91.116.4:4937 , piece_id=MPZ6KR5W5SI2V4L6ERAPSKV6OW4R3HY4BKJ6O3JNDVNJRZXFFN7A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980583411 (527h27m30.851666446s) [noise=true , address=77.90.31.10:4517 , piece_id=6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980937996 (527h27m30.852021041s) [noise=true , address=45.91.116.11:4120 , piece_id=GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981235923 (527h27m30.852318959s) [noise=true , address=145.79.227.7:4295 , piece_id=SRIH4SPOG6ANG4DIYUHM2XAVDDYLPPNE3ANMJI6MEEZJKGQAB56A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981510996 (527h27m30.852594031s) [noise=true , address=145.79.227.49:4338 , piece_id=K22DIQETHGNUQP56JYROWAJW5ZSRHZY4SBHFJ7ZY75ZQXRCOUTNQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981853198 (527h27m30.852936233s) [noise=true , address=145.79.227.51:4410 , piece_id=35UO43XMARP5UYE2R7W4VZY72P76WIBKFGXXVW3NW7SBW44JOK2Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.982462639 (527h27m30.853545694s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.982947583 (527h27m30.854030628s) [noise=true , address=145.79.227.52:4447 , piece_id=XKDSTWXNIZ4L2V4TKCRSMDAKL2KEK5FC5WCNWYEGYHA7MD3SYWGA, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.983327466 (527h27m30.854410501s) [noise=true , address=45.91.116.9:4262 , piece_id=KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:32.605493069 (527h27m32.476576114s) [upload=462637]: Next returned nil segment
Dec 27 14:56:32.605509640 (527h27m32.476592685s) [upload=462637]: Waiting for error group managing segments...
Dec 27 14:56:32.989144216 (527h27m32.860227261s) [noise=true , address=145.79.227.51:4410 , piece_id=35UO43XMARP5UYE2R7W4VZY72P76WIBKFGXXVW3NW7SBW44JOK2Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.990269098 (527h27m32.861352154s) [noise=true , address=145.79.227.52:4447 , piece_id=XKDSTWXNIZ4L2V4TKCRSMDAKL2KEK5FC5WCNWYEGYHA7MD3SYWGA, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.990408243 (527h27m32.861491278s) [noise=true , address=145.79.227.32:4834 , piece_id=E2WAHKFHP2BIBWSMR5UZ6YEOI3EVVS6LBTGBUMSXIGSGXW3B5KWQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.994972074 (527h27m32.866055109s) [noise=true , address=145.79.227.49:4338 , piece_id=K22DIQETHGNUQP56JYROWAJW5ZSRHZY4SBHFJ7ZY75ZQXRCOUTNQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.005702276 (527h27m32.876785341s) [noise=true , address=145.79.227.5:4221 , piece_id=CWIDO7AWOOQZZBPBCDRIJTW3BHJ4ZEJKOFT3XUB6MXEN3NGUAUNQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.007088375 (527h27m32.87817142s) [noise=true , address=145.79.227.11:4730 , piece_id=66IT4MEAZJW6HPXGD4AZY2IXI47VZVJYVTHILSYA4CJPWTUWFK5A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.010248613 (527h27m32.881331648s) [noise=true , address=145.79.227.7:4295 , piece_id=SRIH4SPOG6ANG4DIYUHM2XAVDDYLPPNE3ANMJI6MEEZJKGQAB56A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.015750199 (527h27m32.886833244s) [noise=true , address=145.79.227.45:4231 , piece_id=YDXPUTXI7RDN5FTA7IZOGFBDOLKGFLFCXDOTCQITRCU7NAFCMW5Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.253649410 (527h27m33.124732455s) [noise=true , address=45.91.116.9:4262 , piece_id=KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.269921144 (527h27m33.141004189s) [noise=true , address=45.91.116.11:4120 , piece_id=GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.282601289 (527h27m33.153684334s) [noise=true , address=77.90.31.18:4805 , piece_id=VTKOIICB6NXC5O7RY24NNFZ3MHTC7IRUXI7XNTKOPTN65PDXBRUQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.286531783 (527h27m33.157614818s) [noise=true , address=45.91.116.4:4937 , piece_id=MPZ6KR5W5SI2V4L6ERAPSKV6OW4R3HY4BKJ6O3JNDVNJRZXFFN7A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.299383785 (527h27m33.17046683s) [noise=true , address=77.90.31.10:4517 , piece_id=6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.299406449 (527h27m33.170489494s) [seg_pos={0 0}, upload=462637]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:56:33.299766815 (527h27m33.17084987s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.299776964 (527h27m33.170860009s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300183097 (527h27m33.171266152s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.300192395 (527h27m33.17127543s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300370494 (527h27m33.171453539s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.300436740 (527h27m33.171519775s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300461508 (527h27m33.171544553s) [seg_pos={0 0}, upload=462637]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:56:33.300544585 (527h27m33.17162763s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:56:33.300550718 (527h27m33.171633763s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:56:33.591506157 (527h27m33.462589202s) [upload=462637]: Done uploading segments...
Dec 27 14:56:33.591529251 (527h27m33.462612286s) [upload=462637]: Upload finished. err: <nil>
Duration: 5.663719356s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [............X..X....X.........X.X....X...X.....X..XX.......X] duration:
30.004921146s trace:
https://jaeger.dev.storj.io/trace/2a63206377bc416d
FAIL (2025-12-27 14:57:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q) context deadline exceeded
======LOGS======
Dec 27 14:56:30.727029380 (527h27m30.598112415s) [upload=462637]: Starting upload
Dec 27 14:56:30.727138748 (527h27m30.598221823s) [upload=462637]: Done starting upload
Dec 27 14:56:30.727231635 (527h27m30.59831468s) [upload=462637]: Uploading segments...
Dec 27 14:56:30.727481190 (527h27m30.598564225s) [upload=462637]: Got next segment. Inline: false
Dec 27 14:56:30.727521727 (527h27m30.598604782s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:56:30.727527999 (527h27m30.598611054s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:56:30.978256560 (527h27m30.849339605s) [seg_pos={0 0}, upload=462637]: Begin upload segment...
Dec 27 14:56:30.978308969 (527h27m30.849392004s) [seg_pos={0 0}, upload=462637]: Done begin upload segment.
Dec 27 14:56:30.978341601 (527h27m30.849424646s) [noise=true , address=145.79.227.45:4231 , piece_id=YDXPUTXI7RDN5FTA7IZOGFBDOLKGFLFCXDOTCQITRCU7NAFCMW5Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.978358374 (527h27m30.849441419s) [noise=true , address=145.79.227.11:4730 , piece_id=66IT4MEAZJW6HPXGD4AZY2IXI47VZVJYVTHILSYA4CJPWTUWFK5A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.978981050 (527h27m30.850064085s) [noise=true , address=145.79.227.32:4834 , piece_id=E2WAHKFHP2BIBWSMR5UZ6YEOI3EVVS6LBTGBUMSXIGSGXW3B5KWQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979504376 (527h27m30.850587411s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979586001 (527h27m30.850669036s) [noise=true , address=145.79.227.5:4221 , piece_id=CWIDO7AWOOQZZBPBCDRIJTW3BHJ4ZEJKOFT3XUB6MXEN3NGUAUNQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979878088 (527h27m30.850961123s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.979975422 (527h27m30.851058498s) [noise=true , address=77.90.31.18:4805 , piece_id=VTKOIICB6NXC5O7RY24NNFZ3MHTC7IRUXI7XNTKOPTN65PDXBRUQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980238263 (527h27m30.851321298s) [noise=true , address=45.91.116.4:4937 , piece_id=MPZ6KR5W5SI2V4L6ERAPSKV6OW4R3HY4BKJ6O3JNDVNJRZXFFN7A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980583411 (527h27m30.851666446s) [noise=true , address=77.90.31.10:4517 , piece_id=6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.980937996 (527h27m30.852021041s) [noise=true , address=45.91.116.11:4120 , piece_id=GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981235923 (527h27m30.852318959s) [noise=true , address=145.79.227.7:4295 , piece_id=SRIH4SPOG6ANG4DIYUHM2XAVDDYLPPNE3ANMJI6MEEZJKGQAB56A, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981510996 (527h27m30.852594031s) [noise=true , address=145.79.227.49:4338 , piece_id=K22DIQETHGNUQP56JYROWAJW5ZSRHZY4SBHFJ7ZY75ZQXRCOUTNQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.981853198 (527h27m30.852936233s) [noise=true , address=145.79.227.51:4410 , piece_id=35UO43XMARP5UYE2R7W4VZY72P76WIBKFGXXVW3NW7SBW44JOK2Q, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.982462639 (527h27m30.853545694s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.982947583 (527h27m30.854030628s) [noise=true , address=145.79.227.52:4447 , piece_id=XKDSTWXNIZ4L2V4TKCRSMDAKL2KEK5FC5WCNWYEGYHA7MD3SYWGA, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:30.983327466 (527h27m30.854410501s) [noise=true , address=45.91.116.9:4262 , piece_id=KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ, seg_pos={0 0}, upload=462637]: Uploading piece...
Dec 27 14:56:32.605493069 (527h27m32.476576114s) [upload=462637]: Next returned nil segment
Dec 27 14:56:32.605509640 (527h27m32.476592685s) [upload=462637]: Waiting for error group managing segments...
Dec 27 14:56:32.989144216 (527h27m32.860227261s) [noise=true , address=145.79.227.51:4410 , piece_id=35UO43XMARP5UYE2R7W4VZY72P76WIBKFGXXVW3NW7SBW44JOK2Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.990269098 (527h27m32.861352154s) [noise=true , address=145.79.227.52:4447 , piece_id=XKDSTWXNIZ4L2V4TKCRSMDAKL2KEK5FC5WCNWYEGYHA7MD3SYWGA, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.990408243 (527h27m32.861491278s) [noise=true , address=145.79.227.32:4834 , piece_id=E2WAHKFHP2BIBWSMR5UZ6YEOI3EVVS6LBTGBUMSXIGSGXW3B5KWQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:32.994972074 (527h27m32.866055109s) [noise=true , address=145.79.227.49:4338 , piece_id=K22DIQETHGNUQP56JYROWAJW5ZSRHZY4SBHFJ7ZY75ZQXRCOUTNQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.005702276 (527h27m32.876785341s) [noise=true , address=145.79.227.5:4221 , piece_id=CWIDO7AWOOQZZBPBCDRIJTW3BHJ4ZEJKOFT3XUB6MXEN3NGUAUNQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.007088375 (527h27m32.87817142s) [noise=true , address=145.79.227.11:4730 , piece_id=66IT4MEAZJW6HPXGD4AZY2IXI47VZVJYVTHILSYA4CJPWTUWFK5A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.010248613 (527h27m32.881331648s) [noise=true , address=145.79.227.7:4295 , piece_id=SRIH4SPOG6ANG4DIYUHM2XAVDDYLPPNE3ANMJI6MEEZJKGQAB56A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.015750199 (527h27m32.886833244s) [noise=true , address=145.79.227.45:4231 , piece_id=YDXPUTXI7RDN5FTA7IZOGFBDOLKGFLFCXDOTCQITRCU7NAFCMW5Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.253649410 (527h27m33.124732455s) [noise=true , address=45.91.116.9:4262 , piece_id=KKI4NDEFGMSUAWOR2GLWXX2HY4I7WV6UL2BID4TNENT626B55UVQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.269921144 (527h27m33.141004189s) [noise=true , address=45.91.116.11:4120 , piece_id=GLPYV3GSMHA3RQ736BNNINSEWYDOXRO32H5CYGCRZGINBCNOYO5Q, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.282601289 (527h27m33.153684334s) [noise=true , address=77.90.31.18:4805 , piece_id=VTKOIICB6NXC5O7RY24NNFZ3MHTC7IRUXI7XNTKOPTN65PDXBRUQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.286531783 (527h27m33.157614818s) [noise=true , address=45.91.116.4:4937 , piece_id=MPZ6KR5W5SI2V4L6ERAPSKV6OW4R3HY4BKJ6O3JNDVNJRZXFFN7A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.299383785 (527h27m33.17046683s) [noise=true , address=77.90.31.10:4517 , piece_id=6KNNKFFH7I6Q53NV32O22NBPYQSL2IGV236FTFTYWT7G56H3XH5A, seg_pos={0 0}, upload=462637]: Done uploading piece. err: <nil>
Dec 27 14:56:33.299406449 (527h27m33.170489494s) [seg_pos={0 0}, upload=462637]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:56:33.299766815 (527h27m33.17084987s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.299776964 (527h27m33.170860009s) [noise=true , address=45.91.116.5:4297 , piece_id=LPECN2HCXP3BAZ632XJQVSSUMNRNNRIUDNRVNF2ALZOYPLPQWVWQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300183097 (527h27m33.171266152s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.300192395 (527h27m33.17127543s) [noise=true , address=77.90.31.16:4735 , piece_id=NYQYY35B3JJIROYXGXSEI5GGLFDTNFRVI2HYRVKNDCU3PCPWNNHQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300370494 (527h27m33.171453539s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 14:56:33.300436740 (527h27m33.171519775s) [noise=true , address=145.79.227.30:4762 , piece_id=GFOIBPTELRLMONJQEUJ3CKSG5KOZS4VG4IY753PUKGKCSA6IZHOQ, seg_pos={0 0}, upload=462637]: Long tail cancellation due to optimal threshold
Dec 27 14:56:33.300461508 (527h27m33.171544553s) [seg_pos={0 0}, upload=462637]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:56:33.300544585 (527h27m33.17162763s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:56:33.300550718 (527h27m33.171633763s) [upload=462637]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:56:33.591506157 (527h27m33.462589202s) [upload=462637]: Done uploading segments...
Dec 27 14:56:33.591529251 (527h27m33.462612286s) [upload=462637]: Upload finished. err: <nil>
Duration: 30.004951324s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
17.507576273s trace:
https://jaeger.dev.storj.io/trace/2f00e104e333c43f
FAIL (2025-12-27 14:56:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece...
Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment
Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments...
Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil>
Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces.
Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF
Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold
Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments...
Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil>
Duration: 17.507609716s
core
uplink
eu-select-1
eu1-placement-15
|