Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XX........X..X......X.......XXX...X.....X.......X.XX......XX] duration: 5.001727134s trace: https://jaeger.dev.storj.io/trace/49996d579ff9d89f FAIL (2025-12-26 12:42:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001753254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....................X.] duration: 57.205417425s trace: https://jaeger.dev.storj.io/trace/543b292ca4246ac8 FAIL (2025-12-26 12:41:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0:4406 , piece_id=65LCU4WFRYZWTSUDG6RODGYGLENKB34WQU3OB4DHWWMNIMLEBVRQ, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.782602422 (501h10m35.653685457s) [noise=true , address=64.49.13.169:4711 , piece_id=4N6BZZFTG3AA7CGDQWBHB37HN5MEMSQWB6HJWSRFPJAXKLF3RHPA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.782866786 (501h10m35.653949821s) [noise=true , address=77.90.31.21:4907 , piece_id=ART4A4MOKHGYDZAV5YOUI6LQ3TKXUEDU4LWZHMV45RFMPBWFHPMA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783125628 (501h10m35.654208663s) [noise=true , address=64.49.13.170:4718 , piece_id=ZZPQ6KREVWB2AZFF73QDIQWHH2HF2CYJMLY4L2SEI647JXAH5HNQ, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783401443 (501h10m35.654484478s) [noise=true , address=77.90.31.18:4839 , piece_id=5ODY7L3F7FXFZR3TP257QZ6SAJHTFGULU33DJKD4GKHNB4IHZT7Q, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783680605 (501h10m35.65476364s) [noise=true , address=45.91.116.6:4984 , piece_id=POEGE4DGPETWIT3A7DXEP6JKCMY3D4Y4FVWHEHZKZ63T37RDZXFA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783978082 (501h10m35.655061117s) [noise=true , address=145.79.227.30:4793 , piece_id=6YMNPDAFZJNUDBPMGBCHGSBH4VBJC7AS26DNDDVQUBV6XA7ER5EA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.784319531 (501h10m35.655402566s) [noise=true , address=64.49.13.140:4505 , piece_id=ZABIAWBC73SWYR2PMEHB7SY6UMKVIHGTRSNX3M2C5JIKAFRXVRDA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.786186587 (501h10m35.657269632s) [noise=true , address=45.91.116.13:4167 , piece_id=U43OGD7WJBZM3RYIW62LZ2KOCW43DNB3RPVZCQCHWQ4THZE57DJA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.787231757 (501h10m35.658314792s) [noise=true , address=45.91.116.11:4135 , piece_id=CAEDNOC2MFMNBQ7R4UTLDKDODCYBNI4A5YS6FCAL3XBZETPM5T6A, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.788384482 (501h10m35.659467527s) [noise=true , address=77.90.31.15:4701 , piece_id=PF2VWW5S6OT5BZUVVKWFNYPPE6OC2IIZMBWN2KNUTR4KQ2TYJAIA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.789479597 (501h10m35.660562643s) [noise=true , address=77.90.31.12:4603 , piece_id=S72QE2PUUS3GGUK3LMMBQCB7SAYAOCM5PZLHHNDE7QS72BE5IA4A, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.790703598 (501h10m35.661786633s) [noise=true , address=77.90.31.11:4574 , piece_id=WE3VMKIWDRT64ILCMJFZD3T5B5EFGMZR7JUOY2SRMNX72GL72MTA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.791953067 (501h10m35.663036102s) [noise=true , address=145.79.227.40:4064 , piece_id=G64QSMEF7HEKTRGHYXJPEHJW7I2BJS5EEYCW4CEEE5TGGHDM2AMQ, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.793043063 (501h10m35.664126098s) [noise=true , address=145.79.227.34:4913 , piece_id=B6J7Y5RIJ7N6YZPE5WAUPK6HNWLEAJV3EJMGEE6FDPRIMLLYEMHA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:37.080000795 (501h10m36.95108384s) [upload=439845]: Next returned nil segment Dec 26 12:39:37.080019460 (501h10m36.951102515s) [upload=439845]: Waiting for error group managing segments... Dec 26 12:39:37.416798035 (501h10m37.28788107s) [noise=true , address=64.49.13.140:4505 , piece_id=ZABIAWBC73SWYR2PMEHB7SY6UMKVIHGTRSNX3M2C5JIKAFRXVRDA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.650362483 (501h10m37.521445528s) [noise=true , address=145.79.227.50:4406 , piece_id=65LCU4WFRYZWTSUDG6RODGYGLENKB34WQU3OB4DHWWMNIMLEBVRQ, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.686291559 (501h10m37.557374604s) [noise=true , address=145.79.227.40:4064 , piece_id=G64QSMEF7HEKTRGHYXJPEHJW7I2BJS5EEYCW4CEEE5TGGHDM2AMQ, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.706427240 (501h10m37.577510275s) [noise=true , address=64.49.13.170:4718 , piece_id=ZZPQ6KREVWB2AZFF73QDIQWHH2HF2CYJMLY4L2SEI647JXAH5HNQ, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.715240452 (501h10m37.586323487s) [noise=true , address=145.79.227.13:4830 , piece_id=D6DNOBD4NUPODZ672I7OPTRAOWS4LT6FEWEKPMXT2VNOI4RNVY4A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.881405529 (501h10m37.752488564s) [noise=true , address=77.90.31.12:4603 , piece_id=S72QE2PUUS3GGUK3LMMBQCB7SAYAOCM5PZLHHNDE7QS72BE5IA4A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.906041200 (501h10m37.777124235s) [noise=true , address=45.91.116.13:4167 , piece_id=U43OGD7WJBZM3RYIW62LZ2KOCW43DNB3RPVZCQCHWQ4THZE57DJA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.929044532 (501h10m37.800127567s) [noise=true , address=45.91.116.6:4984 , piece_id=POEGE4DGPETWIT3A7DXEP6JKCMY3D4Y4FVWHEHZKZ63T37RDZXFA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.962294907 (501h10m37.833377942s) [noise=true , address=77.90.31.11:4574 , piece_id=WE3VMKIWDRT64ILCMJFZD3T5B5EFGMZR7JUOY2SRMNX72GL72MTA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.962954713 (501h10m37.834037748s) [noise=true , address=77.90.31.21:4907 , piece_id=ART4A4MOKHGYDZAV5YOUI6LQ3TKXUEDU4LWZHMV45RFMPBWFHPMA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.963941081 (501h10m37.835024126s) [noise=true , address=77.90.31.19:4851 , piece_id=V7HXNQWFYUSOHGC2O4EODCIJL7U4JTEWXP3Q6ZFCLIVEB55NSR3A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.021785728 (501h10m37.892868763s) [noise=true , address=77.90.31.18:4839 , piece_id=5ODY7L3F7FXFZR3TP257QZ6SAJHTFGULU33DJKD4GKHNB4IHZT7Q, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.088646523 (501h10m37.959729558s) [noise=true , address=77.90.31.15:4701 , piece_id=PF2VWW5S6OT5BZUVVKWFNYPPE6OC2IIZMBWN2KNUTR4KQ2TYJAIA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.596753076 (501h10m38.467836121s) [noise=true , address=145.79.227.30:4793 , piece_id=6YMNPDAFZJNUDBPMGBCHGSBH4VBJC7AS26DNDDVQUBV6XA7ER5EA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.596774948 (501h10m38.467857993s) [seg_pos={0 0}, upload=439845]: Segment reached optimal threshold of 14 pieces. Dec 26 12:39:38.596949661 (501h10m38.468032696s) [noise=true , address=145.79.227.34:4913 , piece_id=B6J7Y5RIJ7N6YZPE5WAUPK6HNWLEAJV3EJMGEE6FDPRIMLLYEMHA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:38.596956493 (501h10m38.468039549s) [noise=true , address=145.79.227.34:4913 , piece_id=B6J7Y5RIJ7N6YZPE5WAUPK6HNWLEAJV3EJMGEE6FDPRIMLLYEMHA, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.596972835 (501h10m38.46805587s) [noise=true , address=77.90.31.13:4637 , piece_id=5AVYKZZU6VZ3L73YKVMCD2SDHQSCQY74SV4U63WN3URCTZIECGFA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: write tcp 188.166.225.170:37970->77.90.31.13:4637: use of closed network connection; write tcp 188.166.225.170:37970->77.90.31.13:4637: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37970->77.90.31.13:4637: use of closed network connection Dec 26 12:39:38.596980840 (501h10m38.468063875s) [noise=true , address=77.90.31.13:4637 , piece_id=5AVYKZZU6VZ3L73YKVMCD2SDHQSCQY74SV4U63WN3URCTZIECGFA, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.597043189 (501h10m38.468126224s) [noise=true , address=64.49.13.169:4711 , piece_id=4N6BZZFTG3AA7CGDQWBHB37HN5MEMSQWB6HJWSRFPJAXKLF3RHPA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52480->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52480->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52480->64.49.13.169:4711: use of closed network connection Dec 26 12:39:38.597048679 (501h10m38.468131714s) [noise=true , address=64.49.13.169:4711 , piece_id=4N6BZZFTG3AA7CGDQWBHB37HN5MEMSQWB6HJWSRFPJAXKLF3RHPA, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.597151024 (501h10m38.468234059s) [noise=true , address=45.91.116.11:4135 , piece_id=CAEDNOC2MFMNBQ7R4UTLDKDODCYBNI4A5YS6FCAL3XBZETPM5T6A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:194doyGN13Ha8hsyjrERgTZqdyzFhdTQyiTGQg9rFA7CxvKmkM): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:38.597156174 (501h10m38.468239209s) [noise=true , address=45.91.116.11:4135 , piece_id=CAEDNOC2MFMNBQ7R4UTLDKDODCYBNI4A5YS6FCAL3XBZETPM5T6A, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.597170441 (501h10m38.468253476s) [seg_pos={0 0}, upload=439845]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:39:38.597231136 (501h10m38.468314172s) [upload=439845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:39:38.597235695 (501h10m38.46831873s) [upload=439845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:39:38.885519718 (501h10m38.756602753s) [upload=439845]: Done uploading segments... Dec 26 12:39:38.885540969 (501h10m38.756624014s) [upload=439845]: Upload finished. err: <nil> Duration: 57.205432494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX....X..] duration: 5.726962227s trace: https://jaeger.dev.storj.io/trace/57b611b6b99cab93 FAIL (2025-12-26 12:41:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A) context deadline exceeded ======LOGS====== 12:39:28.198076196 (501h10m28.069159231s) [upload=439843]: Starting upload Dec 26 12:39:28.198144116 (501h10m28.069227161s) [upload=439843]: Done starting upload Dec 26 12:39:28.198313839 (501h10m28.069396874s) [upload=439843]: Uploading segments... Dec 26 12:39:28.198445539 (501h10m28.069528584s) [upload=439843]: Got next segment. Inline: false Dec 26 12:39:28.198502959 (501h10m28.069586014s) [upload=439843]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:39:28.198596678 (501h10m28.069679733s) [upload=439843]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:39:28.443970983 (501h10m28.315054018s) [seg_pos={0 0}, upload=439843]: Begin upload segment... Dec 26 12:39:28.444026277 (501h10m28.315109312s) [seg_pos={0 0}, upload=439843]: Done begin upload segment. Dec 26 12:39:28.444059421 (501h10m28.315142466s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.444136648 (501h10m28.315219694s) [noise=true , address=145.79.227.51:4411 , piece_id=MRGZGSYXMEZCPBSADFXDIJZVU665GD3I66I2ARRBHELA2PNVHKLQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.444772289 (501h10m28.315855334s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.445263063 (501h10m28.316346108s) [noise=true , address=145.79.227.11:4732 , piece_id=DK5RY4323CDKSZVBJHCVUHZVDKLN2UXHHUJIK4CUUZY2H57IEJKA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.445743308 (501h10m28.316826353s) [noise=true , address=45.91.116.5:4297 , piece_id=GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446350353 (501h10m28.317433388s) [noise=true , address=145.79.227.48:4303 , piece_id=6NLWEGQFQTROAQBVWUKQUPHFEGRCMQ6F5RASYTPTAASJQRTX7CJA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446449632 (501h10m28.317532667s) [noise=true , address=145.79.227.3:4150 , piece_id=IRB6T5YL6O4RLJNOWAK4P3GJC7K6EDIT45Q6UMLYV3IOYA53ZP6Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446893578 (501h10m28.317976623s) [noise=true , address=145.79.227.4:4185 , piece_id=E2R57WFKBINSNONKMQM3DUGLHXCC34I6PLIUZC7URSLPNHD3ERIA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447033304 (501h10m28.318116349s) [noise=true , address=145.79.227.19:4078 , piece_id=FAYXMKOEHANZ2OBAQ5C4QIVUYAHD5VLVDADDH27MFK46TYKJ3DZA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447444457 (501h10m28.318527492s) [noise=true , address=145.79.227.45:4230 , piece_id=BARJPINELRMOZSVKVT6QB4CNDX3SDP6QYQAQE6DZAVSQI3KH27VQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447614862 (501h10m28.318697907s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447906547 (501h10m28.318989592s) [noise=true , address=145.79.227.15:4876 , piece_id=4JY63SPOWFVTBRADOWIMXDU2ZWUJEZ2PKVZAQQSMFTZC5ZIQDQNQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447956842 (501h10m28.319039887s) [noise=true , address=145.79.227.6:4258 , piece_id=V7ZE5NZ55ZXUOYTOETONIDK66TD2X7ZIZI3YLBUF2IRSWTNWCH5Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448298603 (501h10m28.319381648s) [noise=true , address=145.79.227.32:4834 , piece_id=N4C7RRHAETJARR6K3WHMRTO6H47EAECN4GLV5VXAKH6ALP7VPJGQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448395488 (501h10m28.319478533s) [noise=true , address=45.91.116.11:4120 , piece_id=NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448689457 (501h10m28.319772502s) [noise=true , address=145.79.227.17:4970 , piece_id=GBPIDPGCTGYLPHUOTTVZX5YEIUP4FKU3P74XEX6COXXNP2NJQW5A, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:30.169234276 (501h10m30.040317331s) [upload=439843]: Next returned nil segment Dec 26 12:39:30.169255256 (501h10m30.040338301s) [upload=439843]: Waiting for error group managing segments... Dec 26 12:39:30.451503545 (501h10m30.32258658s) [noise=true , address=145.79.227.15:4876 , piece_id=4JY63SPOWFVTBRADOWIMXDU2ZWUJEZ2PKVZAQQSMFTZC5ZIQDQNQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.453090999 (501h10m30.324174034s) [noise=true , address=145.79.227.45:4230 , piece_id=BARJPINELRMOZSVKVT6QB4CNDX3SDP6QYQAQE6DZAVSQI3KH27VQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.455518211 (501h10m30.326601256s) [noise=true , address=145.79.227.17:4970 , piece_id=GBPIDPGCTGYLPHUOTTVZX5YEIUP4FKU3P74XEX6COXXNP2NJQW5A, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.456202703 (501h10m30.327285748s) [noise=true , address=145.79.227.51:4411 , piece_id=MRGZGSYXMEZCPBSADFXDIJZVU665GD3I66I2ARRBHELA2PNVHKLQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.458212250 (501h10m30.329295285s) [noise=true , address=145.79.227.48:4303 , piece_id=6NLWEGQFQTROAQBVWUKQUPHFEGRCMQ6F5RASYTPTAASJQRTX7CJA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.460159809 (501h10m30.331242844s) [noise=true , address=145.79.227.6:4258 , piece_id=V7ZE5NZ55ZXUOYTOETONIDK66TD2X7ZIZI3YLBUF2IRSWTNWCH5Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.461956180 (501h10m30.333039225s) [noise=true , address=145.79.227.3:4150 , piece_id=IRB6T5YL6O4RLJNOWAK4P3GJC7K6EDIT45Q6UMLYV3IOYA53ZP6Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.475503877 (501h10m30.346586912s) [noise=true , address=145.79.227.11:4732 , piece_id=DK5RY4323CDKSZVBJHCVUHZVDKLN2UXHHUJIK4CUUZY2H57IEJKA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.478237472 (501h10m30.349320507s) [noise=true , address=145.79.227.32:4834 , piece_id=N4C7RRHAETJARR6K3WHMRTO6H47EAECN4GLV5VXAKH6ALP7VPJGQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.486353847 (501h10m30.357436892s) [noise=true , address=145.79.227.4:4185 , piece_id=E2R57WFKBINSNONKMQM3DUGLHXCC34I6PLIUZC7URSLPNHD3ERIA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.486856153 (501h10m30.357939188s) [noise=true , address=145.79.227.19:4078 , piece_id=FAYXMKOEHANZ2OBAQ5C4QIVUYAHD5VLVDADDH27MFK46TYKJ3DZA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.682432185 (501h10m30.55351522s) [noise=true , address=45.91.116.11:4120 , piece_id=NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.686448442 (501h10m30.557531477s) [noise=true , address=45.91.116.5:4297 , piece_id=GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.686465094 (501h10m30.557548129s) [seg_pos={0 0}, upload=439843]: Segment reached optimal threshold of 13 pieces. Dec 26 12:39:30.686703567 (501h10m30.557786612s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:30.686710271 (501h10m30.557793306s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686749135 (501h10m30.55783217s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:30.686758011 (501h10m30.557841046s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686850558 (501h10m30.557933603s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection Dec 26 12:39:30.686856158 (501h10m30.557939194s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686926182 (501h10m30.558009217s) [seg_pos={0 0}, upload=439843]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:39:30.687005052 (501h10m30.558088087s) [upload=439843]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:39:30.687012255 (501h10m30.55809529s) [upload=439843]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:39:30.968810749 (501h10m30.839893794s) [upload=439843]: Done uploading segments... Dec 26 12:39:30.968828002 (501h10m30.839911037s) [upload=439843]: Upload finished. err: <nil> Duration: 5.726993958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.XX........X..X......X.......XXX...X.....X.......X.XX......X] duration: 5.000783688s trace: https://jaeger.dev.storj.io/trace/17656ad8a5a7d438 FAIL (2025-12-26 12:41:31 UTC) orbiter warning: context deadline exceeded Duration: 5.000809857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX........X..X......X.......XXX...X.....X.......X.XX......] duration: 4.573200193s trace: https://jaeger.dev.storj.io/trace/3947ffbdc793b2a FAIL (2025-12-26 12:40:55 UTC) orbiter warning: context deadline exceeded Duration: 4.573229009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....................X] duration: 1m0.002460318s trace: https://jaeger.dev.storj.io/trace/543b292ca4246ac8 FAIL (2025-12-26 12:40:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0:4406 , piece_id=65LCU4WFRYZWTSUDG6RODGYGLENKB34WQU3OB4DHWWMNIMLEBVRQ, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.782602422 (501h10m35.653685457s) [noise=true , address=64.49.13.169:4711 , piece_id=4N6BZZFTG3AA7CGDQWBHB37HN5MEMSQWB6HJWSRFPJAXKLF3RHPA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.782866786 (501h10m35.653949821s) [noise=true , address=77.90.31.21:4907 , piece_id=ART4A4MOKHGYDZAV5YOUI6LQ3TKXUEDU4LWZHMV45RFMPBWFHPMA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783125628 (501h10m35.654208663s) [noise=true , address=64.49.13.170:4718 , piece_id=ZZPQ6KREVWB2AZFF73QDIQWHH2HF2CYJMLY4L2SEI647JXAH5HNQ, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783401443 (501h10m35.654484478s) [noise=true , address=77.90.31.18:4839 , piece_id=5ODY7L3F7FXFZR3TP257QZ6SAJHTFGULU33DJKD4GKHNB4IHZT7Q, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783680605 (501h10m35.65476364s) [noise=true , address=45.91.116.6:4984 , piece_id=POEGE4DGPETWIT3A7DXEP6JKCMY3D4Y4FVWHEHZKZ63T37RDZXFA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.783978082 (501h10m35.655061117s) [noise=true , address=145.79.227.30:4793 , piece_id=6YMNPDAFZJNUDBPMGBCHGSBH4VBJC7AS26DNDDVQUBV6XA7ER5EA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.784319531 (501h10m35.655402566s) [noise=true , address=64.49.13.140:4505 , piece_id=ZABIAWBC73SWYR2PMEHB7SY6UMKVIHGTRSNX3M2C5JIKAFRXVRDA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.786186587 (501h10m35.657269632s) [noise=true , address=45.91.116.13:4167 , piece_id=U43OGD7WJBZM3RYIW62LZ2KOCW43DNB3RPVZCQCHWQ4THZE57DJA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.787231757 (501h10m35.658314792s) [noise=true , address=45.91.116.11:4135 , piece_id=CAEDNOC2MFMNBQ7R4UTLDKDODCYBNI4A5YS6FCAL3XBZETPM5T6A, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.788384482 (501h10m35.659467527s) [noise=true , address=77.90.31.15:4701 , piece_id=PF2VWW5S6OT5BZUVVKWFNYPPE6OC2IIZMBWN2KNUTR4KQ2TYJAIA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.789479597 (501h10m35.660562643s) [noise=true , address=77.90.31.12:4603 , piece_id=S72QE2PUUS3GGUK3LMMBQCB7SAYAOCM5PZLHHNDE7QS72BE5IA4A, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.790703598 (501h10m35.661786633s) [noise=true , address=77.90.31.11:4574 , piece_id=WE3VMKIWDRT64ILCMJFZD3T5B5EFGMZR7JUOY2SRMNX72GL72MTA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.791953067 (501h10m35.663036102s) [noise=true , address=145.79.227.40:4064 , piece_id=G64QSMEF7HEKTRGHYXJPEHJW7I2BJS5EEYCW4CEEE5TGGHDM2AMQ, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:35.793043063 (501h10m35.664126098s) [noise=true , address=145.79.227.34:4913 , piece_id=B6J7Y5RIJ7N6YZPE5WAUPK6HNWLEAJV3EJMGEE6FDPRIMLLYEMHA, seg_pos={0 0}, upload=439845]: Uploading piece... Dec 26 12:39:37.080000795 (501h10m36.95108384s) [upload=439845]: Next returned nil segment Dec 26 12:39:37.080019460 (501h10m36.951102515s) [upload=439845]: Waiting for error group managing segments... Dec 26 12:39:37.416798035 (501h10m37.28788107s) [noise=true , address=64.49.13.140:4505 , piece_id=ZABIAWBC73SWYR2PMEHB7SY6UMKVIHGTRSNX3M2C5JIKAFRXVRDA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.650362483 (501h10m37.521445528s) [noise=true , address=145.79.227.50:4406 , piece_id=65LCU4WFRYZWTSUDG6RODGYGLENKB34WQU3OB4DHWWMNIMLEBVRQ, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.686291559 (501h10m37.557374604s) [noise=true , address=145.79.227.40:4064 , piece_id=G64QSMEF7HEKTRGHYXJPEHJW7I2BJS5EEYCW4CEEE5TGGHDM2AMQ, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.706427240 (501h10m37.577510275s) [noise=true , address=64.49.13.170:4718 , piece_id=ZZPQ6KREVWB2AZFF73QDIQWHH2HF2CYJMLY4L2SEI647JXAH5HNQ, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.715240452 (501h10m37.586323487s) [noise=true , address=145.79.227.13:4830 , piece_id=D6DNOBD4NUPODZ672I7OPTRAOWS4LT6FEWEKPMXT2VNOI4RNVY4A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.881405529 (501h10m37.752488564s) [noise=true , address=77.90.31.12:4603 , piece_id=S72QE2PUUS3GGUK3LMMBQCB7SAYAOCM5PZLHHNDE7QS72BE5IA4A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.906041200 (501h10m37.777124235s) [noise=true , address=45.91.116.13:4167 , piece_id=U43OGD7WJBZM3RYIW62LZ2KOCW43DNB3RPVZCQCHWQ4THZE57DJA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.929044532 (501h10m37.800127567s) [noise=true , address=45.91.116.6:4984 , piece_id=POEGE4DGPETWIT3A7DXEP6JKCMY3D4Y4FVWHEHZKZ63T37RDZXFA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.962294907 (501h10m37.833377942s) [noise=true , address=77.90.31.11:4574 , piece_id=WE3VMKIWDRT64ILCMJFZD3T5B5EFGMZR7JUOY2SRMNX72GL72MTA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.962954713 (501h10m37.834037748s) [noise=true , address=77.90.31.21:4907 , piece_id=ART4A4MOKHGYDZAV5YOUI6LQ3TKXUEDU4LWZHMV45RFMPBWFHPMA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:37.963941081 (501h10m37.835024126s) [noise=true , address=77.90.31.19:4851 , piece_id=V7HXNQWFYUSOHGC2O4EODCIJL7U4JTEWXP3Q6ZFCLIVEB55NSR3A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.021785728 (501h10m37.892868763s) [noise=true , address=77.90.31.18:4839 , piece_id=5ODY7L3F7FXFZR3TP257QZ6SAJHTFGULU33DJKD4GKHNB4IHZT7Q, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.088646523 (501h10m37.959729558s) [noise=true , address=77.90.31.15:4701 , piece_id=PF2VWW5S6OT5BZUVVKWFNYPPE6OC2IIZMBWN2KNUTR4KQ2TYJAIA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.596753076 (501h10m38.467836121s) [noise=true , address=145.79.227.30:4793 , piece_id=6YMNPDAFZJNUDBPMGBCHGSBH4VBJC7AS26DNDDVQUBV6XA7ER5EA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: <nil> Dec 26 12:39:38.596774948 (501h10m38.467857993s) [seg_pos={0 0}, upload=439845]: Segment reached optimal threshold of 14 pieces. Dec 26 12:39:38.596949661 (501h10m38.468032696s) [noise=true , address=145.79.227.34:4913 , piece_id=B6J7Y5RIJ7N6YZPE5WAUPK6HNWLEAJV3EJMGEE6FDPRIMLLYEMHA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:38.596956493 (501h10m38.468039549s) [noise=true , address=145.79.227.34:4913 , piece_id=B6J7Y5RIJ7N6YZPE5WAUPK6HNWLEAJV3EJMGEE6FDPRIMLLYEMHA, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.596972835 (501h10m38.46805587s) [noise=true , address=77.90.31.13:4637 , piece_id=5AVYKZZU6VZ3L73YKVMCD2SDHQSCQY74SV4U63WN3URCTZIECGFA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: write tcp 188.166.225.170:37970->77.90.31.13:4637: use of closed network connection; write tcp 188.166.225.170:37970->77.90.31.13:4637: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37970->77.90.31.13:4637: use of closed network connection Dec 26 12:39:38.596980840 (501h10m38.468063875s) [noise=true , address=77.90.31.13:4637 , piece_id=5AVYKZZU6VZ3L73YKVMCD2SDHQSCQY74SV4U63WN3URCTZIECGFA, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.597043189 (501h10m38.468126224s) [noise=true , address=64.49.13.169:4711 , piece_id=4N6BZZFTG3AA7CGDQWBHB37HN5MEMSQWB6HJWSRFPJAXKLF3RHPA, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52480->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52480->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52480->64.49.13.169:4711: use of closed network connection Dec 26 12:39:38.597048679 (501h10m38.468131714s) [noise=true , address=64.49.13.169:4711 , piece_id=4N6BZZFTG3AA7CGDQWBHB37HN5MEMSQWB6HJWSRFPJAXKLF3RHPA, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.597151024 (501h10m38.468234059s) [noise=true , address=45.91.116.11:4135 , piece_id=CAEDNOC2MFMNBQ7R4UTLDKDODCYBNI4A5YS6FCAL3XBZETPM5T6A, seg_pos={0 0}, upload=439845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:194doyGN13Ha8hsyjrERgTZqdyzFhdTQyiTGQg9rFA7CxvKmkM): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:38.597156174 (501h10m38.468239209s) [noise=true , address=45.91.116.11:4135 , piece_id=CAEDNOC2MFMNBQ7R4UTLDKDODCYBNI4A5YS6FCAL3XBZETPM5T6A, seg_pos={0 0}, upload=439845]: Long tail cancellation due to optimal threshold Dec 26 12:39:38.597170441 (501h10m38.468253476s) [seg_pos={0 0}, upload=439845]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:39:38.597231136 (501h10m38.468314172s) [upload=439845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:39:38.597235695 (501h10m38.46831873s) [upload=439845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:39:38.885519718 (501h10m38.756602753s) [upload=439845]: Done uploading segments... Dec 26 12:39:38.885540969 (501h10m38.756624014s) [upload=439845]: Upload finished. err: <nil> Duration: 1m0.002471048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX....X.] duration: 5.801586176s trace: https://jaeger.dev.storj.io/trace/57b611b6b99cab93 FAIL (2025-12-26 12:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A) context deadline exceeded ======LOGS====== 12:39:28.198076196 (501h10m28.069159231s) [upload=439843]: Starting upload Dec 26 12:39:28.198144116 (501h10m28.069227161s) [upload=439843]: Done starting upload Dec 26 12:39:28.198313839 (501h10m28.069396874s) [upload=439843]: Uploading segments... Dec 26 12:39:28.198445539 (501h10m28.069528584s) [upload=439843]: Got next segment. Inline: false Dec 26 12:39:28.198502959 (501h10m28.069586014s) [upload=439843]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:39:28.198596678 (501h10m28.069679733s) [upload=439843]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:39:28.443970983 (501h10m28.315054018s) [seg_pos={0 0}, upload=439843]: Begin upload segment... Dec 26 12:39:28.444026277 (501h10m28.315109312s) [seg_pos={0 0}, upload=439843]: Done begin upload segment. Dec 26 12:39:28.444059421 (501h10m28.315142466s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.444136648 (501h10m28.315219694s) [noise=true , address=145.79.227.51:4411 , piece_id=MRGZGSYXMEZCPBSADFXDIJZVU665GD3I66I2ARRBHELA2PNVHKLQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.444772289 (501h10m28.315855334s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.445263063 (501h10m28.316346108s) [noise=true , address=145.79.227.11:4732 , piece_id=DK5RY4323CDKSZVBJHCVUHZVDKLN2UXHHUJIK4CUUZY2H57IEJKA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.445743308 (501h10m28.316826353s) [noise=true , address=45.91.116.5:4297 , piece_id=GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446350353 (501h10m28.317433388s) [noise=true , address=145.79.227.48:4303 , piece_id=6NLWEGQFQTROAQBVWUKQUPHFEGRCMQ6F5RASYTPTAASJQRTX7CJA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446449632 (501h10m28.317532667s) [noise=true , address=145.79.227.3:4150 , piece_id=IRB6T5YL6O4RLJNOWAK4P3GJC7K6EDIT45Q6UMLYV3IOYA53ZP6Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446893578 (501h10m28.317976623s) [noise=true , address=145.79.227.4:4185 , piece_id=E2R57WFKBINSNONKMQM3DUGLHXCC34I6PLIUZC7URSLPNHD3ERIA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447033304 (501h10m28.318116349s) [noise=true , address=145.79.227.19:4078 , piece_id=FAYXMKOEHANZ2OBAQ5C4QIVUYAHD5VLVDADDH27MFK46TYKJ3DZA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447444457 (501h10m28.318527492s) [noise=true , address=145.79.227.45:4230 , piece_id=BARJPINELRMOZSVKVT6QB4CNDX3SDP6QYQAQE6DZAVSQI3KH27VQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447614862 (501h10m28.318697907s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447906547 (501h10m28.318989592s) [noise=true , address=145.79.227.15:4876 , piece_id=4JY63SPOWFVTBRADOWIMXDU2ZWUJEZ2PKVZAQQSMFTZC5ZIQDQNQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447956842 (501h10m28.319039887s) [noise=true , address=145.79.227.6:4258 , piece_id=V7ZE5NZ55ZXUOYTOETONIDK66TD2X7ZIZI3YLBUF2IRSWTNWCH5Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448298603 (501h10m28.319381648s) [noise=true , address=145.79.227.32:4834 , piece_id=N4C7RRHAETJARR6K3WHMRTO6H47EAECN4GLV5VXAKH6ALP7VPJGQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448395488 (501h10m28.319478533s) [noise=true , address=45.91.116.11:4120 , piece_id=NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448689457 (501h10m28.319772502s) [noise=true , address=145.79.227.17:4970 , piece_id=GBPIDPGCTGYLPHUOTTVZX5YEIUP4FKU3P74XEX6COXXNP2NJQW5A, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:30.169234276 (501h10m30.040317331s) [upload=439843]: Next returned nil segment Dec 26 12:39:30.169255256 (501h10m30.040338301s) [upload=439843]: Waiting for error group managing segments... Dec 26 12:39:30.451503545 (501h10m30.32258658s) [noise=true , address=145.79.227.15:4876 , piece_id=4JY63SPOWFVTBRADOWIMXDU2ZWUJEZ2PKVZAQQSMFTZC5ZIQDQNQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.453090999 (501h10m30.324174034s) [noise=true , address=145.79.227.45:4230 , piece_id=BARJPINELRMOZSVKVT6QB4CNDX3SDP6QYQAQE6DZAVSQI3KH27VQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.455518211 (501h10m30.326601256s) [noise=true , address=145.79.227.17:4970 , piece_id=GBPIDPGCTGYLPHUOTTVZX5YEIUP4FKU3P74XEX6COXXNP2NJQW5A, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.456202703 (501h10m30.327285748s) [noise=true , address=145.79.227.51:4411 , piece_id=MRGZGSYXMEZCPBSADFXDIJZVU665GD3I66I2ARRBHELA2PNVHKLQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.458212250 (501h10m30.329295285s) [noise=true , address=145.79.227.48:4303 , piece_id=6NLWEGQFQTROAQBVWUKQUPHFEGRCMQ6F5RASYTPTAASJQRTX7CJA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.460159809 (501h10m30.331242844s) [noise=true , address=145.79.227.6:4258 , piece_id=V7ZE5NZ55ZXUOYTOETONIDK66TD2X7ZIZI3YLBUF2IRSWTNWCH5Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.461956180 (501h10m30.333039225s) [noise=true , address=145.79.227.3:4150 , piece_id=IRB6T5YL6O4RLJNOWAK4P3GJC7K6EDIT45Q6UMLYV3IOYA53ZP6Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.475503877 (501h10m30.346586912s) [noise=true , address=145.79.227.11:4732 , piece_id=DK5RY4323CDKSZVBJHCVUHZVDKLN2UXHHUJIK4CUUZY2H57IEJKA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.478237472 (501h10m30.349320507s) [noise=true , address=145.79.227.32:4834 , piece_id=N4C7RRHAETJARR6K3WHMRTO6H47EAECN4GLV5VXAKH6ALP7VPJGQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.486353847 (501h10m30.357436892s) [noise=true , address=145.79.227.4:4185 , piece_id=E2R57WFKBINSNONKMQM3DUGLHXCC34I6PLIUZC7URSLPNHD3ERIA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.486856153 (501h10m30.357939188s) [noise=true , address=145.79.227.19:4078 , piece_id=FAYXMKOEHANZ2OBAQ5C4QIVUYAHD5VLVDADDH27MFK46TYKJ3DZA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.682432185 (501h10m30.55351522s) [noise=true , address=45.91.116.11:4120 , piece_id=NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.686448442 (501h10m30.557531477s) [noise=true , address=45.91.116.5:4297 , piece_id=GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.686465094 (501h10m30.557548129s) [seg_pos={0 0}, upload=439843]: Segment reached optimal threshold of 13 pieces. Dec 26 12:39:30.686703567 (501h10m30.557786612s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:30.686710271 (501h10m30.557793306s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686749135 (501h10m30.55783217s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:30.686758011 (501h10m30.557841046s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686850558 (501h10m30.557933603s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection Dec 26 12:39:30.686856158 (501h10m30.557939194s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686926182 (501h10m30.558009217s) [seg_pos={0 0}, upload=439843]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:39:30.687005052 (501h10m30.558088087s) [upload=439843]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:39:30.687012255 (501h10m30.55809529s) [upload=439843]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:39:30.968810749 (501h10m30.839893794s) [upload=439843]: Done uploading segments... Dec 26 12:39:30.968828002 (501h10m30.839911037s) [upload=439843]: Upload finished. err: <nil> Duration: 5.801611765s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XX........X..X......X.......XXX...X.....X.......X.XX.....] duration: 4.248708966s trace: https://jaeger.dev.storj.io/trace/3947ffbdc793b2a FAIL (2025-12-26 12:40:12 UTC) orbiter warning: context deadline exceeded Duration: 4.248736909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX....X] duration: 30.003095074s trace: https://jaeger.dev.storj.io/trace/57b611b6b99cab93 FAIL (2025-12-26 12:39:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A) context deadline exceeded ======LOGS====== 12:39:28.198076196 (501h10m28.069159231s) [upload=439843]: Starting upload Dec 26 12:39:28.198144116 (501h10m28.069227161s) [upload=439843]: Done starting upload Dec 26 12:39:28.198313839 (501h10m28.069396874s) [upload=439843]: Uploading segments... Dec 26 12:39:28.198445539 (501h10m28.069528584s) [upload=439843]: Got next segment. Inline: false Dec 26 12:39:28.198502959 (501h10m28.069586014s) [upload=439843]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:39:28.198596678 (501h10m28.069679733s) [upload=439843]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:39:28.443970983 (501h10m28.315054018s) [seg_pos={0 0}, upload=439843]: Begin upload segment... Dec 26 12:39:28.444026277 (501h10m28.315109312s) [seg_pos={0 0}, upload=439843]: Done begin upload segment. Dec 26 12:39:28.444059421 (501h10m28.315142466s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.444136648 (501h10m28.315219694s) [noise=true , address=145.79.227.51:4411 , piece_id=MRGZGSYXMEZCPBSADFXDIJZVU665GD3I66I2ARRBHELA2PNVHKLQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.444772289 (501h10m28.315855334s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.445263063 (501h10m28.316346108s) [noise=true , address=145.79.227.11:4732 , piece_id=DK5RY4323CDKSZVBJHCVUHZVDKLN2UXHHUJIK4CUUZY2H57IEJKA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.445743308 (501h10m28.316826353s) [noise=true , address=45.91.116.5:4297 , piece_id=GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446350353 (501h10m28.317433388s) [noise=true , address=145.79.227.48:4303 , piece_id=6NLWEGQFQTROAQBVWUKQUPHFEGRCMQ6F5RASYTPTAASJQRTX7CJA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446449632 (501h10m28.317532667s) [noise=true , address=145.79.227.3:4150 , piece_id=IRB6T5YL6O4RLJNOWAK4P3GJC7K6EDIT45Q6UMLYV3IOYA53ZP6Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.446893578 (501h10m28.317976623s) [noise=true , address=145.79.227.4:4185 , piece_id=E2R57WFKBINSNONKMQM3DUGLHXCC34I6PLIUZC7URSLPNHD3ERIA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447033304 (501h10m28.318116349s) [noise=true , address=145.79.227.19:4078 , piece_id=FAYXMKOEHANZ2OBAQ5C4QIVUYAHD5VLVDADDH27MFK46TYKJ3DZA, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447444457 (501h10m28.318527492s) [noise=true , address=145.79.227.45:4230 , piece_id=BARJPINELRMOZSVKVT6QB4CNDX3SDP6QYQAQE6DZAVSQI3KH27VQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447614862 (501h10m28.318697907s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447906547 (501h10m28.318989592s) [noise=true , address=145.79.227.15:4876 , piece_id=4JY63SPOWFVTBRADOWIMXDU2ZWUJEZ2PKVZAQQSMFTZC5ZIQDQNQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.447956842 (501h10m28.319039887s) [noise=true , address=145.79.227.6:4258 , piece_id=V7ZE5NZ55ZXUOYTOETONIDK66TD2X7ZIZI3YLBUF2IRSWTNWCH5Q, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448298603 (501h10m28.319381648s) [noise=true , address=145.79.227.32:4834 , piece_id=N4C7RRHAETJARR6K3WHMRTO6H47EAECN4GLV5VXAKH6ALP7VPJGQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448395488 (501h10m28.319478533s) [noise=true , address=45.91.116.11:4120 , piece_id=NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:28.448689457 (501h10m28.319772502s) [noise=true , address=145.79.227.17:4970 , piece_id=GBPIDPGCTGYLPHUOTTVZX5YEIUP4FKU3P74XEX6COXXNP2NJQW5A, seg_pos={0 0}, upload=439843]: Uploading piece... Dec 26 12:39:30.169234276 (501h10m30.040317331s) [upload=439843]: Next returned nil segment Dec 26 12:39:30.169255256 (501h10m30.040338301s) [upload=439843]: Waiting for error group managing segments... Dec 26 12:39:30.451503545 (501h10m30.32258658s) [noise=true , address=145.79.227.15:4876 , piece_id=4JY63SPOWFVTBRADOWIMXDU2ZWUJEZ2PKVZAQQSMFTZC5ZIQDQNQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.453090999 (501h10m30.324174034s) [noise=true , address=145.79.227.45:4230 , piece_id=BARJPINELRMOZSVKVT6QB4CNDX3SDP6QYQAQE6DZAVSQI3KH27VQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.455518211 (501h10m30.326601256s) [noise=true , address=145.79.227.17:4970 , piece_id=GBPIDPGCTGYLPHUOTTVZX5YEIUP4FKU3P74XEX6COXXNP2NJQW5A, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.456202703 (501h10m30.327285748s) [noise=true , address=145.79.227.51:4411 , piece_id=MRGZGSYXMEZCPBSADFXDIJZVU665GD3I66I2ARRBHELA2PNVHKLQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.458212250 (501h10m30.329295285s) [noise=true , address=145.79.227.48:4303 , piece_id=6NLWEGQFQTROAQBVWUKQUPHFEGRCMQ6F5RASYTPTAASJQRTX7CJA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.460159809 (501h10m30.331242844s) [noise=true , address=145.79.227.6:4258 , piece_id=V7ZE5NZ55ZXUOYTOETONIDK66TD2X7ZIZI3YLBUF2IRSWTNWCH5Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.461956180 (501h10m30.333039225s) [noise=true , address=145.79.227.3:4150 , piece_id=IRB6T5YL6O4RLJNOWAK4P3GJC7K6EDIT45Q6UMLYV3IOYA53ZP6Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.475503877 (501h10m30.346586912s) [noise=true , address=145.79.227.11:4732 , piece_id=DK5RY4323CDKSZVBJHCVUHZVDKLN2UXHHUJIK4CUUZY2H57IEJKA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.478237472 (501h10m30.349320507s) [noise=true , address=145.79.227.32:4834 , piece_id=N4C7RRHAETJARR6K3WHMRTO6H47EAECN4GLV5VXAKH6ALP7VPJGQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.486353847 (501h10m30.357436892s) [noise=true , address=145.79.227.4:4185 , piece_id=E2R57WFKBINSNONKMQM3DUGLHXCC34I6PLIUZC7URSLPNHD3ERIA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.486856153 (501h10m30.357939188s) [noise=true , address=145.79.227.19:4078 , piece_id=FAYXMKOEHANZ2OBAQ5C4QIVUYAHD5VLVDADDH27MFK46TYKJ3DZA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.682432185 (501h10m30.55351522s) [noise=true , address=45.91.116.11:4120 , piece_id=NYZC7ZWVZEOZP7VZRBFLJVYNV5CHFIVDXHYBCOVQJ5BL62JJ4DRQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.686448442 (501h10m30.557531477s) [noise=true , address=45.91.116.5:4297 , piece_id=GZR3PG6IDTAZINCQF6KAGSGZ3BGN63EV5RVDHMY362RTMMSELV2A, seg_pos={0 0}, upload=439843]: Done uploading piece. err: <nil> Dec 26 12:39:30.686465094 (501h10m30.557548129s) [seg_pos={0 0}, upload=439843]: Segment reached optimal threshold of 13 pieces. Dec 26 12:39:30.686703567 (501h10m30.557786612s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:30.686710271 (501h10m30.557793306s) [noise=true , address=145.79.227.33:4870 , piece_id=TTO3MHZ5YM6LUBDHUTYXKKSUDJH6RXUOJTAJCLBAGBQISGKUIPTA, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686749135 (501h10m30.55783217s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 12:39:30.686758011 (501h10m30.557841046s) [noise=true , address=77.90.31.10:4520 , piece_id=UMMBR5463VFJJ37D6M2BGRSFTBZ3LLQXCY4WVFDUSG5OU4L7DZZQ, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686850558 (501h10m30.557933603s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52770->45.91.116.14:4190: use of closed network connection Dec 26 12:39:30.686856158 (501h10m30.557939194s) [noise=true , address=45.91.116.14:4190 , piece_id=Y3CULPP2W4E2G6ZSULY5EGCDUTLIY2JEDEWUVJJDM35KVAJNQQ2Q, seg_pos={0 0}, upload=439843]: Long tail cancellation due to optimal threshold Dec 26 12:39:30.686926182 (501h10m30.558009217s) [seg_pos={0 0}, upload=439843]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:39:30.687005052 (501h10m30.558088087s) [upload=439843]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:39:30.687012255 (501h10m30.55809529s) [upload=439843]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:39:30.968810749 (501h10m30.839893794s) [upload=439843]: Done uploading segments... Dec 26 12:39:30.968828002 (501h10m30.839911037s) [upload=439843]: Upload finished. err: <nil> Duration: 30.003120372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 54.256725241s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:39:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 54.256752722s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....XX........X..X......X.......XXX...X.....X.......X.XX....] duration: 3.489279409s trace: https://jaeger.dev.storj.io/trace/3947ffbdc793b2a FAIL (2025-12-26 12:39:19 UTC) orbiter warning: context deadline exceeded Duration: 3.48935386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX....] duration: 8.397527421s trace: https://jaeger.dev.storj.io/trace/3bb7ea215515348 FAIL (2025-12-26 12:39:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA) context deadline exceeded ======LOGS====== Dec 26 12:36:03.474425148 (501h7m3.345508193s) [upload=439796]: Starting upload Dec 26 12:36:03.474514117 (501h7m3.345597162s) [upload=439796]: Done starting upload Dec 26 12:36:03.474669673 (501h7m3.345752708s) [upload=439796]: Uploading segments... Dec 26 12:36:03.474896876 (501h7m3.345979921s) [upload=439796]: Got next segment. Inline: false Dec 26 12:36:03.474952592 (501h7m3.346035637s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:36:03.474958914 (501h7m3.346041959s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:36:03.721735219 (501h7m3.592818275s) [seg_pos={0 0}, upload=439796]: Begin upload segment... Dec 26 12:36:03.721793069 (501h7m3.592876114s) [seg_pos={0 0}, upload=439796]: Done begin upload segment. Dec 26 12:36:03.721876368 (501h7m3.592959413s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722115434 (501h7m3.593198489s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722807951 (501h7m3.593890996s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723203675 (501h7m3.594286721s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723582938 (501h7m3.594665983s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724022494 (501h7m3.595105539s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724420983 (501h7m3.595504039s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724821836 (501h7m3.595904881s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.725306469 (501h7m3.596389514s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726123505 (501h7m3.597206551s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726730531 (501h7m3.597813576s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727301458 (501h7m3.598384503s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727811589 (501h7m3.598894634s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.728912937 (501h7m3.599995982s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.729478995 (501h7m3.60056204s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.730423783 (501h7m3.601506839s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:05.443531310 (501h7m5.314614355s) [upload=439796]: Next returned nil segment Dec 26 12:36:05.443545638 (501h7m5.314628683s) [upload=439796]: Waiting for error group managing segments... Dec 26 12:36:05.730910059 (501h7m5.601993094s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.732256824 (501h7m5.603339869s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.734468536 (501h7m5.605551571s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735006690 (501h7m5.606089735s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735070412 (501h7m5.606153457s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755154044 (501h7m5.626237089s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755857253 (501h7m5.626940288s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.756659120 (501h7m5.627742165s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.757229085 (501h7m5.62831213s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.761842350 (501h7m5.632925385s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.762294311 (501h7m5.633377356s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.978314170 (501h7m5.849397215s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986270531 (501h7m5.857353576s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986288284 (501h7m5.857371319s) [seg_pos={0 0}, upload=439796]: Segment reached optimal threshold of 13 pieces. Dec 26 12:36:05.986546115 (501h7m5.85762916s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986553279 (501h7m5.857636314s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986739012 (501h7m5.857822078s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986818253 (501h7m5.857901288s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986997715 (501h7m5.85808075s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44726->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:36:05.987006242 (501h7m5.858089297s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.987027692 (501h7m5.858110737s) [seg_pos={0 0}, upload=439796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:36:05.987090281 (501h7m5.858173326s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:36:05.987095000 (501h7m5.858178045s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:36:06.275010181 (501h7m6.146093217s) [upload=439796]: Done uploading segments... Dec 26 12:36:06.275029047 (501h7m6.146112092s) [upload=439796]: Upload finished. err: <nil> Duration: 8.397555706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....XX........X..X......X.......XXX...X.....X.......X.XX...] duration: 3.624366889s trace: https://jaeger.dev.storj.io/trace/3947ffbdc793b2a FAIL (2025-12-26 12:38:48 UTC) orbiter warning: context deadline exceeded Duration: 3.624391526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 40.887169102s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:38:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 40.887185042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX...] duration: 5.989113901s trace: https://jaeger.dev.storj.io/trace/3bb7ea215515348 FAIL (2025-12-26 12:38:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA) context deadline exceeded ======LOGS====== Dec 26 12:36:03.474425148 (501h7m3.345508193s) [upload=439796]: Starting upload Dec 26 12:36:03.474514117 (501h7m3.345597162s) [upload=439796]: Done starting upload Dec 26 12:36:03.474669673 (501h7m3.345752708s) [upload=439796]: Uploading segments... Dec 26 12:36:03.474896876 (501h7m3.345979921s) [upload=439796]: Got next segment. Inline: false Dec 26 12:36:03.474952592 (501h7m3.346035637s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:36:03.474958914 (501h7m3.346041959s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:36:03.721735219 (501h7m3.592818275s) [seg_pos={0 0}, upload=439796]: Begin upload segment... Dec 26 12:36:03.721793069 (501h7m3.592876114s) [seg_pos={0 0}, upload=439796]: Done begin upload segment. Dec 26 12:36:03.721876368 (501h7m3.592959413s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722115434 (501h7m3.593198489s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722807951 (501h7m3.593890996s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723203675 (501h7m3.594286721s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723582938 (501h7m3.594665983s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724022494 (501h7m3.595105539s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724420983 (501h7m3.595504039s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724821836 (501h7m3.595904881s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.725306469 (501h7m3.596389514s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726123505 (501h7m3.597206551s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726730531 (501h7m3.597813576s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727301458 (501h7m3.598384503s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727811589 (501h7m3.598894634s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.728912937 (501h7m3.599995982s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.729478995 (501h7m3.60056204s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.730423783 (501h7m3.601506839s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:05.443531310 (501h7m5.314614355s) [upload=439796]: Next returned nil segment Dec 26 12:36:05.443545638 (501h7m5.314628683s) [upload=439796]: Waiting for error group managing segments... Dec 26 12:36:05.730910059 (501h7m5.601993094s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.732256824 (501h7m5.603339869s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.734468536 (501h7m5.605551571s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735006690 (501h7m5.606089735s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735070412 (501h7m5.606153457s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755154044 (501h7m5.626237089s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755857253 (501h7m5.626940288s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.756659120 (501h7m5.627742165s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.757229085 (501h7m5.62831213s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.761842350 (501h7m5.632925385s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.762294311 (501h7m5.633377356s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.978314170 (501h7m5.849397215s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986270531 (501h7m5.857353576s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986288284 (501h7m5.857371319s) [seg_pos={0 0}, upload=439796]: Segment reached optimal threshold of 13 pieces. Dec 26 12:36:05.986546115 (501h7m5.85762916s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986553279 (501h7m5.857636314s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986739012 (501h7m5.857822078s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986818253 (501h7m5.857901288s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986997715 (501h7m5.85808075s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44726->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:36:05.987006242 (501h7m5.858089297s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.987027692 (501h7m5.858110737s) [seg_pos={0 0}, upload=439796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:36:05.987090281 (501h7m5.858173326s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:36:05.987095000 (501h7m5.858178045s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:36:06.275010181 (501h7m6.146093217s) [upload=439796]: Done uploading segments... Dec 26 12:36:06.275029047 (501h7m6.146112092s) [upload=439796]: Upload finished. err: <nil> Duration: 5.989145322s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......XX........X..X......X.......XXX...X.....X.......X.XX..] duration: 4.421023133s trace: https://jaeger.dev.storj.io/trace/3947ffbdc793b2a FAIL (2025-12-26 12:38:03 UTC) orbiter warning: context deadline exceeded Duration: 4.421051806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX..] duration: 5.66849265s trace: https://jaeger.dev.storj.io/trace/3bb7ea215515348 FAIL (2025-12-26 12:37:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA) context deadline exceeded ======LOGS====== Dec 26 12:36:03.474425148 (501h7m3.345508193s) [upload=439796]: Starting upload Dec 26 12:36:03.474514117 (501h7m3.345597162s) [upload=439796]: Done starting upload Dec 26 12:36:03.474669673 (501h7m3.345752708s) [upload=439796]: Uploading segments... Dec 26 12:36:03.474896876 (501h7m3.345979921s) [upload=439796]: Got next segment. Inline: false Dec 26 12:36:03.474952592 (501h7m3.346035637s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:36:03.474958914 (501h7m3.346041959s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:36:03.721735219 (501h7m3.592818275s) [seg_pos={0 0}, upload=439796]: Begin upload segment... Dec 26 12:36:03.721793069 (501h7m3.592876114s) [seg_pos={0 0}, upload=439796]: Done begin upload segment. Dec 26 12:36:03.721876368 (501h7m3.592959413s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722115434 (501h7m3.593198489s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722807951 (501h7m3.593890996s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723203675 (501h7m3.594286721s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723582938 (501h7m3.594665983s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724022494 (501h7m3.595105539s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724420983 (501h7m3.595504039s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724821836 (501h7m3.595904881s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.725306469 (501h7m3.596389514s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726123505 (501h7m3.597206551s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726730531 (501h7m3.597813576s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727301458 (501h7m3.598384503s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727811589 (501h7m3.598894634s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.728912937 (501h7m3.599995982s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.729478995 (501h7m3.60056204s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.730423783 (501h7m3.601506839s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:05.443531310 (501h7m5.314614355s) [upload=439796]: Next returned nil segment Dec 26 12:36:05.443545638 (501h7m5.314628683s) [upload=439796]: Waiting for error group managing segments... Dec 26 12:36:05.730910059 (501h7m5.601993094s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.732256824 (501h7m5.603339869s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.734468536 (501h7m5.605551571s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735006690 (501h7m5.606089735s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735070412 (501h7m5.606153457s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755154044 (501h7m5.626237089s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755857253 (501h7m5.626940288s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.756659120 (501h7m5.627742165s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.757229085 (501h7m5.62831213s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.761842350 (501h7m5.632925385s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.762294311 (501h7m5.633377356s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.978314170 (501h7m5.849397215s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986270531 (501h7m5.857353576s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986288284 (501h7m5.857371319s) [seg_pos={0 0}, upload=439796]: Segment reached optimal threshold of 13 pieces. Dec 26 12:36:05.986546115 (501h7m5.85762916s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986553279 (501h7m5.857636314s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986739012 (501h7m5.857822078s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986818253 (501h7m5.857901288s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986997715 (501h7m5.85808075s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44726->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:36:05.987006242 (501h7m5.858089297s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.987027692 (501h7m5.858110737s) [seg_pos={0 0}, upload=439796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:36:05.987090281 (501h7m5.858173326s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:36:05.987095000 (501h7m5.858178045s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:36:06.275010181 (501h7m6.146093217s) [upload=439796]: Done uploading segments... Dec 26 12:36:06.275029047 (501h7m6.146112092s) [upload=439796]: Upload finished. err: <nil> Duration: 5.668523149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 51.74527049s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:37:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 51.745299755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX.] duration: 6.110853714s trace: https://jaeger.dev.storj.io/trace/3bb7ea215515348 FAIL (2025-12-26 12:37:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA) context deadline exceeded ======LOGS====== Dec 26 12:36:03.474425148 (501h7m3.345508193s) [upload=439796]: Starting upload Dec 26 12:36:03.474514117 (501h7m3.345597162s) [upload=439796]: Done starting upload Dec 26 12:36:03.474669673 (501h7m3.345752708s) [upload=439796]: Uploading segments... Dec 26 12:36:03.474896876 (501h7m3.345979921s) [upload=439796]: Got next segment. Inline: false Dec 26 12:36:03.474952592 (501h7m3.346035637s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:36:03.474958914 (501h7m3.346041959s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:36:03.721735219 (501h7m3.592818275s) [seg_pos={0 0}, upload=439796]: Begin upload segment... Dec 26 12:36:03.721793069 (501h7m3.592876114s) [seg_pos={0 0}, upload=439796]: Done begin upload segment. Dec 26 12:36:03.721876368 (501h7m3.592959413s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722115434 (501h7m3.593198489s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722807951 (501h7m3.593890996s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723203675 (501h7m3.594286721s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723582938 (501h7m3.594665983s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724022494 (501h7m3.595105539s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724420983 (501h7m3.595504039s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724821836 (501h7m3.595904881s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.725306469 (501h7m3.596389514s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726123505 (501h7m3.597206551s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726730531 (501h7m3.597813576s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727301458 (501h7m3.598384503s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727811589 (501h7m3.598894634s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.728912937 (501h7m3.599995982s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.729478995 (501h7m3.60056204s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.730423783 (501h7m3.601506839s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:05.443531310 (501h7m5.314614355s) [upload=439796]: Next returned nil segment Dec 26 12:36:05.443545638 (501h7m5.314628683s) [upload=439796]: Waiting for error group managing segments... Dec 26 12:36:05.730910059 (501h7m5.601993094s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.732256824 (501h7m5.603339869s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.734468536 (501h7m5.605551571s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735006690 (501h7m5.606089735s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735070412 (501h7m5.606153457s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755154044 (501h7m5.626237089s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755857253 (501h7m5.626940288s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.756659120 (501h7m5.627742165s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.757229085 (501h7m5.62831213s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.761842350 (501h7m5.632925385s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.762294311 (501h7m5.633377356s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.978314170 (501h7m5.849397215s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986270531 (501h7m5.857353576s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986288284 (501h7m5.857371319s) [seg_pos={0 0}, upload=439796]: Segment reached optimal threshold of 13 pieces. Dec 26 12:36:05.986546115 (501h7m5.85762916s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986553279 (501h7m5.857636314s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986739012 (501h7m5.857822078s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986818253 (501h7m5.857901288s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986997715 (501h7m5.85808075s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44726->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:36:05.987006242 (501h7m5.858089297s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.987027692 (501h7m5.858110737s) [seg_pos={0 0}, upload=439796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:36:05.987090281 (501h7m5.858173326s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:36:05.987095000 (501h7m5.858178045s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:36:06.275010181 (501h7m6.146093217s) [upload=439796]: Done uploading segments... Dec 26 12:36:06.275029047 (501h7m6.146112092s) [upload=439796]: Upload finished. err: <nil> Duration: 6.110882288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......XX........X..X......X.......XXX...X.....X.......X.XX.] duration: 5.000788306s trace: https://jaeger.dev.storj.io/trace/3947ffbdc793b2a FAIL (2025-12-26 12:36:58 UTC) orbiter warning: context deadline exceeded Duration: 5.000816648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX] duration: 30.002273655s trace: https://jaeger.dev.storj.io/trace/3bb7ea215515348 FAIL (2025-12-26 12:36:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA) context deadline exceeded ======LOGS====== Dec 26 12:36:03.474425148 (501h7m3.345508193s) [upload=439796]: Starting upload Dec 26 12:36:03.474514117 (501h7m3.345597162s) [upload=439796]: Done starting upload Dec 26 12:36:03.474669673 (501h7m3.345752708s) [upload=439796]: Uploading segments... Dec 26 12:36:03.474896876 (501h7m3.345979921s) [upload=439796]: Got next segment. Inline: false Dec 26 12:36:03.474952592 (501h7m3.346035637s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:36:03.474958914 (501h7m3.346041959s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:36:03.721735219 (501h7m3.592818275s) [seg_pos={0 0}, upload=439796]: Begin upload segment... Dec 26 12:36:03.721793069 (501h7m3.592876114s) [seg_pos={0 0}, upload=439796]: Done begin upload segment. Dec 26 12:36:03.721876368 (501h7m3.592959413s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722115434 (501h7m3.593198489s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.722807951 (501h7m3.593890996s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723203675 (501h7m3.594286721s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.723582938 (501h7m3.594665983s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724022494 (501h7m3.595105539s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724420983 (501h7m3.595504039s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.724821836 (501h7m3.595904881s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.725306469 (501h7m3.596389514s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726123505 (501h7m3.597206551s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.726730531 (501h7m3.597813576s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727301458 (501h7m3.598384503s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.727811589 (501h7m3.598894634s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.728912937 (501h7m3.599995982s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.729478995 (501h7m3.60056204s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:03.730423783 (501h7m3.601506839s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Uploading piece... Dec 26 12:36:05.443531310 (501h7m5.314614355s) [upload=439796]: Next returned nil segment Dec 26 12:36:05.443545638 (501h7m5.314628683s) [upload=439796]: Waiting for error group managing segments... Dec 26 12:36:05.730910059 (501h7m5.601993094s) [noise=true , address=145.79.227.39:4013 , piece_id=YHJ7KWYYWDQXCYOCQTKZVHMCZNPXYHXIUAUM4CJINPK4FLCTHZYA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.732256824 (501h7m5.603339869s) [noise=true , address=145.79.227.11:4731 , piece_id=SIOT2CYEXABTXWVTQSVQ2LBQ7V4JT4T3DNLYLUNG5UVU3II5VOVA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.734468536 (501h7m5.605551571s) [noise=true , address=145.79.227.3:4149 , piece_id=TO5BJCOZF3XR3YGAIOFH3YDL2BOCOT7TORFWK4WDRA3MJ2BAUVHQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735006690 (501h7m5.606089735s) [noise=true , address=145.79.227.2:4115 , piece_id=USROYYHDATT72MCS4XZJUASJWCKFXNF73E7UWR7YXHWKDO56HAWA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.735070412 (501h7m5.606153457s) [noise=true , address=145.79.227.46:4601 , piece_id=URFTWURKVDKDBXFYN7PUI65LPFTP6YM7BAM4B5YBBFJNFI7CZ3PQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755154044 (501h7m5.626237089s) [noise=true , address=145.79.227.43:4158 , piece_id=XB5CEMJBYYTTLK3TC3QX42HUST3MIBYBL63LK4EOQDRU6GDQR3CQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.755857253 (501h7m5.626940288s) [noise=true , address=145.79.227.36:4519 , piece_id=7QDCMYR3COHLCUN4UDR5JQHSBZZXEYCPROPTRBH4JQDTALA3FJOQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.756659120 (501h7m5.627742165s) [noise=true , address=145.79.227.41:4086 , piece_id=MUJIAEY2AGBKWRYAXJ4OJ7UDIYHHNXXF34QO2OKCDAJH4VILA7XA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.757229085 (501h7m5.62831213s) [noise=true , address=145.79.227.38:4977 , piece_id=FTPXISZEKHRH7XQUSF27KZ4ER6TB3LPERDBNN5K72XV6GQPTP3FQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.761842350 (501h7m5.632925385s) [noise=true , address=145.79.227.37:4566 , piece_id=HRA3LC5OZJPL5YGK5VAWC3IHO5QUT5JX3EVKK2SIMVH7AXQKWDUQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.762294311 (501h7m5.633377356s) [noise=true , address=145.79.227.15:4876 , piece_id=JYLV2EHFLTTX4ZKP2L25QFGVYJYHQOTDQTEXRKUAE5VBLBKZNAJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.978314170 (501h7m5.849397215s) [noise=true , address=45.91.116.10:4084 , piece_id=T6BEYTK2XVIFLWYR5RCLAER6FUQA5RWDE2OQW3TAPRYZJ4ZUF5MQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986270531 (501h7m5.857353576s) [noise=true , address=77.90.31.14:4664 , piece_id=DN3NXIUJWWVPETMGJQ43ET73UVCKIJJBER2SICBKUDIR2OOXM2FA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: <nil> Dec 26 12:36:05.986288284 (501h7m5.857371319s) [seg_pos={0 0}, upload=439796]: Segment reached optimal threshold of 13 pieces. Dec 26 12:36:05.986546115 (501h7m5.85762916s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986553279 (501h7m5.857636314s) [noise=true , address=145.79.227.40:4051 , piece_id=CDUE2WKGI3XFQVJY7H4LLXTIRLN247DBJD3GAFHTLEGON43FC6OA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986739012 (501h7m5.857822078s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 26 12:36:05.986818253 (501h7m5.857901288s) [noise=true , address=77.90.31.11:4553 , piece_id=ZIGHAYWSOSXIUNVM76BVYLIVNTOHP653WCVJ6MFQJB63JR7IBIEQ, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.986997715 (501h7m5.85808075s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44726->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:36:05.987006242 (501h7m5.858089297s) [noise=true , address=45.91.116.14:4190 , piece_id=N2M67SBWH4HIP2UBR6ZHY23Z2JCAPD2YN3UTPVSP34JBZXJP3OJA, seg_pos={0 0}, upload=439796]: Long tail cancellation due to optimal threshold Dec 26 12:36:05.987027692 (501h7m5.858110737s) [seg_pos={0 0}, upload=439796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:36:05.987090281 (501h7m5.858173326s) [upload=439796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:36:05.987095000 (501h7m5.858178045s) [upload=439796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:36:06.275010181 (501h7m6.146093217s) [upload=439796]: Done uploading segments... Dec 26 12:36:06.275029047 (501h7m6.146112092s) [upload=439796]: Upload finished. err: <nil> Duration: 30.002300106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 54.803389586s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:36:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 54.803445451s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [........XX........X..X......X.......XXX...X.....X.......X.XX] duration: 5.000895418s trace: https://jaeger.dev.storj.io/trace/3947ffbdc793b2a FAIL (2025-12-26 12:35:50 UTC) orbiter warning: context deadline exceeded Duration: 5.000913143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [........X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....X] duration: 30.003261075s trace: https://jaeger.dev.storj.io/trace/35f3fa786ae0e28c FAIL (2025-12-26 12:35:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: VQ4TFKZVOBAHGY4ZMAVUDOIWRSTJZOPAUUXLVNXFELJIP5ZGFF7A) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 7QMWKQA722OTBWCUJ7EHZWZVDZ6RNZDFT6MJTDYHNHW5PAJ6XKTA) context deadline exceeded ======LOGS====== Dec 26 12:35:17.143441446 (501h6m17.014524491s) [upload=439784]: Starting upload Dec 26 12:35:17.143533911 (501h6m17.014616958s) [upload=439784]: Done starting upload Dec 26 12:35:17.143635565 (501h6m17.0147186s) [upload=439784]: Uploading segments... Dec 26 12:35:17.143779139 (501h6m17.014862184s) [upload=439784]: Got next segment. Inline: false Dec 26 12:35:17.143825858 (501h6m17.014908893s) [upload=439784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:35:17.143831378 (501h6m17.014914423s) [upload=439784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:35:17.388620469 (501h6m17.259703504s) [seg_pos={0 0}, upload=439784]: Begin upload segment... Dec 26 12:35:17.388684872 (501h6m17.259767917s) [seg_pos={0 0}, upload=439784]: Done begin upload segment. Dec 26 12:35:17.388728686 (501h6m17.259811721s) [noise=true , address=145.79.227.52:4446 , piece_id=7FNZMLSI45TPN4NB4WIQE6ORQEJNMOC4P2WFLWZ5Q7ZJH6TTZYRQ, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.388766558 (501h6m17.259849603s) [noise=true , address=145.79.227.47:4267 , piece_id=PEE7RTMUS5OQQBP3FI3W67SLLDRAU5IC7A24VPMO7MD3PPD4TITA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.389229730 (501h6m17.260312775s) [noise=true , address=145.79.227.2:4115 , piece_id=CAAUFS57FISLTO44TYYSAPVLC6TXTEIAIEOLCAMRGXJNKW3UYGCQ, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.389661952 (501h6m17.260744997s) [noise=true , address=77.90.31.14:4664 , piece_id=R6CM3Y3T566LYEVAJ72CDFR62V4PDRMSHHR6O5VU26OGGHCESS2A, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.390120907 (501h6m17.261203952s) [noise=true , address=145.79.227.22:4511 , piece_id=YZ5YUDQTPVZ6HRKM332GESF765HNSXDVCLTSZSFX37T4PBB3NKNQ, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.390535647 (501h6m17.261618692s) [noise=true , address=145.79.227.35:4481 , piece_id=WFSOEWTS5VZJ4QRPMB2S72D54P52PGCEHKTAPKJXWTTFSXVQS2EQ, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.391122413 (501h6m17.262205458s) [noise=true , address=145.79.227.39:4015 , piece_id=LMV3BGJLS5GCA4Z5ZTE3BN4RV3ZVCFP5I6KPTPSOIG4VIPKLQ4BA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.391616305 (501h6m17.26269938s) [noise=true , address=145.79.227.11:4731 , piece_id=F625YLVSIKP4BURI5RBHAY2EFXSQL2BM5TEO4MDVRT6ZLZEHADYA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.392200156 (501h6m17.263283201s) [noise=true , address=145.79.227.34:4907 , piece_id=ZKOS6OP46XJIL2L3YIFP53BZ7T4DPMFQ6QWF2HCMC2ZAO5LXTOZQ, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.393510631 (501h6m17.264593686s) [noise=true , address=45.91.116.8:4047 , piece_id=7QMWKQA722OTBWCUJ7EHZWZVDZ6RNZDFT6MJTDYHNHW5PAJ6XKTA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.393592186 (501h6m17.264675231s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVESJIC77IGNBKHJJ35GPTFAEOOKHJDP2GZBQAFD2DAQJDVH33GA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.395332620 (501h6m17.266415665s) [noise=true , address=145.79.227.38:4978 , piece_id=5OZI6BAO2O52V57BIVT5O2ATXBUL2I5AB2YE66G2DXGQBRB6C6DA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.396772042 (501h6m17.267855087s) [noise=true , address=145.79.227.33:4869 , piece_id=NPI6U5KXCBUXEY7YC7BZOI2CDYX4LDJZ6TCERB46EJ3Y5USKTWOA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.396855280 (501h6m17.267938325s) [noise=true , address=45.91.116.14:4189 , piece_id=VQ4TFKZVOBAHGY4ZMAVUDOIWRSTJZOPAUUXLVNXFELJIP5ZGFF7A, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.398413618 (501h6m17.269496663s) [noise=true , address=145.79.227.3:4149 , piece_id=TTO7XFFET4JNQ77TOJ62QJAREYSMKISNSLWVOFCUWICYU4ALUTOA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:17.399683485 (501h6m17.27076652s) [noise=true , address=45.91.116.11:4118 , piece_id=6SRT2XDBLVR5UYW3Q73IM3VWOEMK77IZS36NCZJU3HHLCO7WH5IA, seg_pos={0 0}, upload=439784]: Uploading piece... Dec 26 12:35:19.069162185 (501h6m18.94024523s) [upload=439784]: Next returned nil segment Dec 26 12:35:19.069178967 (501h6m18.940262012s) [upload=439784]: Waiting for error group managing segments... Dec 26 12:35:19.388606022 (501h6m19.259689067s) [noise=true , address=145.79.227.2:4115 , piece_id=CAAUFS57FISLTO44TYYSAPVLC6TXTEIAIEOLCAMRGXJNKW3UYGCQ, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.402526799 (501h6m19.273609844s) [noise=true , address=145.79.227.35:4481 , piece_id=WFSOEWTS5VZJ4QRPMB2S72D54P52PGCEHKTAPKJXWTTFSXVQS2EQ, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.404980242 (501h6m19.276063287s) [noise=true , address=145.79.227.22:4511 , piece_id=YZ5YUDQTPVZ6HRKM332GESF765HNSXDVCLTSZSFX37T4PBB3NKNQ, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.405038592 (501h6m19.276121637s) [noise=true , address=145.79.227.3:4149 , piece_id=TTO7XFFET4JNQ77TOJ62QJAREYSMKISNSLWVOFCUWICYU4ALUTOA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.405528937 (501h6m19.276611982s) [noise=true , address=145.79.227.52:4446 , piece_id=7FNZMLSI45TPN4NB4WIQE6ORQEJNMOC4P2WFLWZ5Q7ZJH6TTZYRQ, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.407458040 (501h6m19.278541085s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVESJIC77IGNBKHJJ35GPTFAEOOKHJDP2GZBQAFD2DAQJDVH33GA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.408411715 (501h6m19.27949476s) [noise=true , address=145.79.227.39:4015 , piece_id=LMV3BGJLS5GCA4Z5ZTE3BN4RV3ZVCFP5I6KPTPSOIG4VIPKLQ4BA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.412586956 (501h6m19.283670001s) [noise=true , address=145.79.227.33:4869 , piece_id=NPI6U5KXCBUXEY7YC7BZOI2CDYX4LDJZ6TCERB46EJ3Y5USKTWOA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.413191618 (501h6m19.284274653s) [noise=true , address=145.79.227.11:4731 , piece_id=F625YLVSIKP4BURI5RBHAY2EFXSQL2BM5TEO4MDVRT6ZLZEHADYA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.434056307 (501h6m19.305139362s) [noise=true , address=145.79.227.38:4978 , piece_id=5OZI6BAO2O52V57BIVT5O2ATXBUL2I5AB2YE66G2DXGQBRB6C6DA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.452589118 (501h6m19.323672163s) [noise=true , address=145.79.227.47:4267 , piece_id=PEE7RTMUS5OQQBP3FI3W67SLLDRAU5IC7A24VPMO7MD3PPD4TITA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.708356212 (501h6m19.579439257s) [noise=true , address=45.91.116.14:4189 , piece_id=VQ4TFKZVOBAHGY4ZMAVUDOIWRSTJZOPAUUXLVNXFELJIP5ZGFF7A, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.758478193 (501h6m19.629561228s) [noise=true , address=45.91.116.8:4047 , piece_id=7QMWKQA722OTBWCUJ7EHZWZVDZ6RNZDFT6MJTDYHNHW5PAJ6XKTA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: <nil> Dec 26 12:35:19.758500496 (501h6m19.629583541s) [seg_pos={0 0}, upload=439784]: Segment reached optimal threshold of 13 pieces. Dec 26 12:35:19.758908383 (501h6m19.629991418s) [noise=true , address=77.90.31.14:4664 , piece_id=R6CM3Y3T566LYEVAJ72CDFR62V4PDRMSHHR6O5VU26OGGHCESS2A, seg_pos={0 0}, upload=439784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 12:35:19.758918051 (501h6m19.630001096s) [noise=true , address=77.90.31.14:4664 , piece_id=R6CM3Y3T566LYEVAJ72CDFR62V4PDRMSHHR6O5VU26OGGHCESS2A, seg_pos={0 0}, upload=439784]: Long tail cancellation due to optimal threshold Dec 26 12:35:19.759094206 (501h6m19.630177251s) [noise=true , address=45.91.116.11:4118 , piece_id=6SRT2XDBLVR5UYW3Q73IM3VWOEMK77IZS36NCZJU3HHLCO7WH5IA, seg_pos={0 0}, upload=439784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 12:35:19.759107912 (501h6m19.630190957s) [noise=true , address=45.91.116.11:4118 , piece_id=6SRT2XDBLVR5UYW3Q73IM3VWOEMK77IZS36NCZJU3HHLCO7WH5IA, seg_pos={0 0}, upload=439784]: Long tail cancellation due to optimal threshold Dec 26 12:35:19.759150333 (501h6m19.630233389s) [noise=true , address=145.79.227.34:4907 , piece_id=ZKOS6OP46XJIL2L3YIFP53BZ7T4DPMFQ6QWF2HCMC2ZAO5LXTOZQ, seg_pos={0 0}, upload=439784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 12:35:19.759174379 (501h6m19.630257424s) [noise=true , address=145.79.227.34:4907 , piece_id=ZKOS6OP46XJIL2L3YIFP53BZ7T4DPMFQ6QWF2HCMC2ZAO5LXTOZQ, seg_pos={0 0}, upload=439784]: Long tail cancellation due to optimal threshold Dec 26 12:35:19.759210638 (501h6m19.630293673s) [seg_pos={0 0}, upload=439784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:35:19.759399899 (501h6m19.630482944s) [upload=439784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:35:19.759409286 (501h6m19.630492331s) [upload=439784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:35:20.055982623 (501h6m19.927065678s) [upload=439784]: Done uploading segments... Dec 26 12:35:20.056010787 (501h6m19.927093832s) [upload=439784]: Upload finished. err: <nil> Duration: 30.003286543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 43.152205504s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 43.15223471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.........XX........X..X......X.......XXX...X.....X.......X.X] duration: 5.001076284s trace: https://jaeger.dev.storj.io/trace/6e9493e1e4a4d251 FAIL (2025-12-26 12:34:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001108835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..........XX........X..X......X.......XXX...X.....X.......X.] duration: 3.703801776s trace: https://jaeger.dev.storj.io/trace/281d036d0a9c0eed FAIL (2025-12-26 12:34:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.703857904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....] duration: 8.985454937s trace: https://jaeger.dev.storj.io/trace/6c5354216884d937 FAIL (2025-12-26 12:34:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q) context deadline exceeded ======LOGS====== Dec 26 12:31:49.650592416 (501h2m49.521675461s) [upload=439732]: Starting upload Dec 26 12:31:49.650777509 (501h2m49.521860554s) [upload=439732]: Done starting upload Dec 26 12:31:49.651025611 (501h2m49.522108656s) [upload=439732]: Uploading segments... Dec 26 12:31:49.651092849 (501h2m49.522175904s) [upload=439732]: Got next segment. Inline: false Dec 26 12:31:49.651193190 (501h2m49.522276225s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:31:49.651200093 (501h2m49.522283138s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:31:49.896845486 (501h2m49.767928521s) [seg_pos={0 0}, upload=439732]: Begin upload segment... Dec 26 12:31:49.896926641 (501h2m49.768009686s) [seg_pos={0 0}, upload=439732]: Done begin upload segment. Dec 26 12:31:49.896938233 (501h2m49.768021268s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897002916 (501h2m49.768085961s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897947805 (501h2m49.76903085s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898020754 (501h2m49.769103789s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898367094 (501h2m49.769450139s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898786382 (501h2m49.769869427s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899110258 (501h2m49.770193303s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899182826 (501h2m49.770265872s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899835660 (501h2m49.770918705s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900376831 (501h2m49.771459876s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900983485 (501h2m49.772066531s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901445685 (501h2m49.77252873s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901987968 (501h2m49.773071013s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.902454897 (501h2m49.773537942s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903023881 (501h2m49.774106926s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903141916 (501h2m49.774224961s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:51.618042576 (501h2m51.489125621s) [upload=439732]: Next returned nil segment Dec 26 12:31:51.618062223 (501h2m51.489145258s) [upload=439732]: Waiting for error group managing segments... Dec 26 12:31:51.899859815 (501h2m51.77094285s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.904980976 (501h2m51.776064021s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.905001846 (501h2m51.776084882s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.919874727 (501h2m51.790957772s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.921036388 (501h2m51.792119433s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.924479284 (501h2m51.795562329s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.926029998 (501h2m51.797113043s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.927476752 (501h2m51.798559807s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933634898 (501h2m51.804717933s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933840370 (501h2m51.804923415s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.126336676 (501h2m51.997419721s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.267758879 (501h2m52.138841924s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858149345 (501h2m52.72923238s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858169524 (501h2m52.729252569s) [seg_pos={0 0}, upload=439732]: Segment reached optimal threshold of 13 pieces. Dec 26 12:31:52.858437404 (501h2m52.729520439s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858445700 (501h2m52.729528745s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858575958 (501h2m52.729659003s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858687290 (501h2m52.729770335s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858851242 (501h2m52.729934287s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858860570 (501h2m52.729943615s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858882661 (501h2m52.729965706s) [seg_pos={0 0}, upload=439732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:31:52.858968074 (501h2m52.73005111s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:31:52.858974116 (501h2m52.730057161s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:31:53.152007553 (501h2m53.023090588s) [upload=439732]: Done uploading segments... Dec 26 12:31:53.152027951 (501h2m53.023110986s) [upload=439732]: Upload finished. err: <nil> Duration: 8.985482229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 38.322169119s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:34:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 38.322199107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...........XX........X..X......X.......XXX...X.....X.......X] duration: 5.001531269s trace: https://jaeger.dev.storj.io/trace/281d036d0a9c0eed FAIL (2025-12-26 12:34:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00155773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX...] duration: 5.952258701s trace: https://jaeger.dev.storj.io/trace/6c5354216884d937 FAIL (2025-12-26 12:33:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q) context deadline exceeded ======LOGS====== Dec 26 12:31:49.650592416 (501h2m49.521675461s) [upload=439732]: Starting upload Dec 26 12:31:49.650777509 (501h2m49.521860554s) [upload=439732]: Done starting upload Dec 26 12:31:49.651025611 (501h2m49.522108656s) [upload=439732]: Uploading segments... Dec 26 12:31:49.651092849 (501h2m49.522175904s) [upload=439732]: Got next segment. Inline: false Dec 26 12:31:49.651193190 (501h2m49.522276225s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:31:49.651200093 (501h2m49.522283138s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:31:49.896845486 (501h2m49.767928521s) [seg_pos={0 0}, upload=439732]: Begin upload segment... Dec 26 12:31:49.896926641 (501h2m49.768009686s) [seg_pos={0 0}, upload=439732]: Done begin upload segment. Dec 26 12:31:49.896938233 (501h2m49.768021268s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897002916 (501h2m49.768085961s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897947805 (501h2m49.76903085s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898020754 (501h2m49.769103789s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898367094 (501h2m49.769450139s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898786382 (501h2m49.769869427s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899110258 (501h2m49.770193303s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899182826 (501h2m49.770265872s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899835660 (501h2m49.770918705s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900376831 (501h2m49.771459876s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900983485 (501h2m49.772066531s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901445685 (501h2m49.77252873s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901987968 (501h2m49.773071013s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.902454897 (501h2m49.773537942s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903023881 (501h2m49.774106926s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903141916 (501h2m49.774224961s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:51.618042576 (501h2m51.489125621s) [upload=439732]: Next returned nil segment Dec 26 12:31:51.618062223 (501h2m51.489145258s) [upload=439732]: Waiting for error group managing segments... Dec 26 12:31:51.899859815 (501h2m51.77094285s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.904980976 (501h2m51.776064021s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.905001846 (501h2m51.776084882s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.919874727 (501h2m51.790957772s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.921036388 (501h2m51.792119433s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.924479284 (501h2m51.795562329s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.926029998 (501h2m51.797113043s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.927476752 (501h2m51.798559807s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933634898 (501h2m51.804717933s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933840370 (501h2m51.804923415s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.126336676 (501h2m51.997419721s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.267758879 (501h2m52.138841924s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858149345 (501h2m52.72923238s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858169524 (501h2m52.729252569s) [seg_pos={0 0}, upload=439732]: Segment reached optimal threshold of 13 pieces. Dec 26 12:31:52.858437404 (501h2m52.729520439s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858445700 (501h2m52.729528745s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858575958 (501h2m52.729659003s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858687290 (501h2m52.729770335s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858851242 (501h2m52.729934287s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858860570 (501h2m52.729943615s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858882661 (501h2m52.729965706s) [seg_pos={0 0}, upload=439732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:31:52.858968074 (501h2m52.73005111s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:31:52.858974116 (501h2m52.730057161s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:31:53.152007553 (501h2m53.023090588s) [upload=439732]: Done uploading segments... Dec 26 12:31:53.152027951 (501h2m53.023110986s) [upload=439732]: Upload finished. err: <nil> Duration: 5.952290511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............XX........X..X......X.......XXX...X.....X.......] duration: 3.467238809s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:33:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.467264708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 54.59634644s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:33:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 54.59636248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX..] duration: 5.794944898s trace: https://jaeger.dev.storj.io/trace/6c5354216884d937 FAIL (2025-12-26 12:33:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q) context deadline exceeded ======LOGS====== Dec 26 12:31:49.650592416 (501h2m49.521675461s) [upload=439732]: Starting upload Dec 26 12:31:49.650777509 (501h2m49.521860554s) [upload=439732]: Done starting upload Dec 26 12:31:49.651025611 (501h2m49.522108656s) [upload=439732]: Uploading segments... Dec 26 12:31:49.651092849 (501h2m49.522175904s) [upload=439732]: Got next segment. Inline: false Dec 26 12:31:49.651193190 (501h2m49.522276225s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:31:49.651200093 (501h2m49.522283138s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:31:49.896845486 (501h2m49.767928521s) [seg_pos={0 0}, upload=439732]: Begin upload segment... Dec 26 12:31:49.896926641 (501h2m49.768009686s) [seg_pos={0 0}, upload=439732]: Done begin upload segment. Dec 26 12:31:49.896938233 (501h2m49.768021268s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897002916 (501h2m49.768085961s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897947805 (501h2m49.76903085s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898020754 (501h2m49.769103789s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898367094 (501h2m49.769450139s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898786382 (501h2m49.769869427s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899110258 (501h2m49.770193303s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899182826 (501h2m49.770265872s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899835660 (501h2m49.770918705s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900376831 (501h2m49.771459876s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900983485 (501h2m49.772066531s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901445685 (501h2m49.77252873s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901987968 (501h2m49.773071013s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.902454897 (501h2m49.773537942s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903023881 (501h2m49.774106926s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903141916 (501h2m49.774224961s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:51.618042576 (501h2m51.489125621s) [upload=439732]: Next returned nil segment Dec 26 12:31:51.618062223 (501h2m51.489145258s) [upload=439732]: Waiting for error group managing segments... Dec 26 12:31:51.899859815 (501h2m51.77094285s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.904980976 (501h2m51.776064021s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.905001846 (501h2m51.776084882s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.919874727 (501h2m51.790957772s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.921036388 (501h2m51.792119433s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.924479284 (501h2m51.795562329s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.926029998 (501h2m51.797113043s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.927476752 (501h2m51.798559807s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933634898 (501h2m51.804717933s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933840370 (501h2m51.804923415s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.126336676 (501h2m51.997419721s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.267758879 (501h2m52.138841924s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858149345 (501h2m52.72923238s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858169524 (501h2m52.729252569s) [seg_pos={0 0}, upload=439732]: Segment reached optimal threshold of 13 pieces. Dec 26 12:31:52.858437404 (501h2m52.729520439s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858445700 (501h2m52.729528745s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858575958 (501h2m52.729659003s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858687290 (501h2m52.729770335s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858851242 (501h2m52.729934287s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858860570 (501h2m52.729943615s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858882661 (501h2m52.729965706s) [seg_pos={0 0}, upload=439732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:31:52.858968074 (501h2m52.73005111s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:31:52.858974116 (501h2m52.730057161s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:31:53.152007553 (501h2m53.023090588s) [upload=439732]: Done uploading segments... Dec 26 12:31:53.152027951 (501h2m53.023110986s) [upload=439732]: Upload finished. err: <nil> Duration: 5.794974504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX.] duration: 5.998448325s trace: https://jaeger.dev.storj.io/trace/6c5354216884d937 FAIL (2025-12-26 12:32:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q) context deadline exceeded ======LOGS====== Dec 26 12:31:49.650592416 (501h2m49.521675461s) [upload=439732]: Starting upload Dec 26 12:31:49.650777509 (501h2m49.521860554s) [upload=439732]: Done starting upload Dec 26 12:31:49.651025611 (501h2m49.522108656s) [upload=439732]: Uploading segments... Dec 26 12:31:49.651092849 (501h2m49.522175904s) [upload=439732]: Got next segment. Inline: false Dec 26 12:31:49.651193190 (501h2m49.522276225s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:31:49.651200093 (501h2m49.522283138s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:31:49.896845486 (501h2m49.767928521s) [seg_pos={0 0}, upload=439732]: Begin upload segment... Dec 26 12:31:49.896926641 (501h2m49.768009686s) [seg_pos={0 0}, upload=439732]: Done begin upload segment. Dec 26 12:31:49.896938233 (501h2m49.768021268s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897002916 (501h2m49.768085961s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897947805 (501h2m49.76903085s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898020754 (501h2m49.769103789s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898367094 (501h2m49.769450139s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898786382 (501h2m49.769869427s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899110258 (501h2m49.770193303s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899182826 (501h2m49.770265872s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899835660 (501h2m49.770918705s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900376831 (501h2m49.771459876s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900983485 (501h2m49.772066531s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901445685 (501h2m49.77252873s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901987968 (501h2m49.773071013s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.902454897 (501h2m49.773537942s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903023881 (501h2m49.774106926s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903141916 (501h2m49.774224961s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:51.618042576 (501h2m51.489125621s) [upload=439732]: Next returned nil segment Dec 26 12:31:51.618062223 (501h2m51.489145258s) [upload=439732]: Waiting for error group managing segments... Dec 26 12:31:51.899859815 (501h2m51.77094285s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.904980976 (501h2m51.776064021s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.905001846 (501h2m51.776084882s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.919874727 (501h2m51.790957772s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.921036388 (501h2m51.792119433s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.924479284 (501h2m51.795562329s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.926029998 (501h2m51.797113043s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.927476752 (501h2m51.798559807s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933634898 (501h2m51.804717933s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933840370 (501h2m51.804923415s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.126336676 (501h2m51.997419721s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.267758879 (501h2m52.138841924s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858149345 (501h2m52.72923238s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858169524 (501h2m52.729252569s) [seg_pos={0 0}, upload=439732]: Segment reached optimal threshold of 13 pieces. Dec 26 12:31:52.858437404 (501h2m52.729520439s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858445700 (501h2m52.729528745s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858575958 (501h2m52.729659003s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858687290 (501h2m52.729770335s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858851242 (501h2m52.729934287s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858860570 (501h2m52.729943615s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858882661 (501h2m52.729965706s) [seg_pos={0 0}, upload=439732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:31:52.858968074 (501h2m52.73005111s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:31:52.858974116 (501h2m52.730057161s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:31:53.152007553 (501h2m53.023090588s) [upload=439732]: Done uploading segments... Dec 26 12:31:53.152027951 (501h2m53.023110986s) [upload=439732]: Upload finished. err: <nil> Duration: 5.998474324s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X............XX........X..X......X.......XXX...X.....X......] duration: 5.001209136s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:32:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001234375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 58.0559884s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:32:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 58.056018538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....XXX] duration: 30.003559503s trace: https://jaeger.dev.storj.io/trace/6c5354216884d937 FAIL (2025-12-26 12:32:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q) context deadline exceeded ======LOGS====== Dec 26 12:31:49.650592416 (501h2m49.521675461s) [upload=439732]: Starting upload Dec 26 12:31:49.650777509 (501h2m49.521860554s) [upload=439732]: Done starting upload Dec 26 12:31:49.651025611 (501h2m49.522108656s) [upload=439732]: Uploading segments... Dec 26 12:31:49.651092849 (501h2m49.522175904s) [upload=439732]: Got next segment. Inline: false Dec 26 12:31:49.651193190 (501h2m49.522276225s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:31:49.651200093 (501h2m49.522283138s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:31:49.896845486 (501h2m49.767928521s) [seg_pos={0 0}, upload=439732]: Begin upload segment... Dec 26 12:31:49.896926641 (501h2m49.768009686s) [seg_pos={0 0}, upload=439732]: Done begin upload segment. Dec 26 12:31:49.896938233 (501h2m49.768021268s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897002916 (501h2m49.768085961s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.897947805 (501h2m49.76903085s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898020754 (501h2m49.769103789s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898367094 (501h2m49.769450139s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.898786382 (501h2m49.769869427s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899110258 (501h2m49.770193303s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899182826 (501h2m49.770265872s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.899835660 (501h2m49.770918705s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900376831 (501h2m49.771459876s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.900983485 (501h2m49.772066531s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901445685 (501h2m49.77252873s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.901987968 (501h2m49.773071013s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.902454897 (501h2m49.773537942s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903023881 (501h2m49.774106926s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:49.903141916 (501h2m49.774224961s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Uploading piece... Dec 26 12:31:51.618042576 (501h2m51.489125621s) [upload=439732]: Next returned nil segment Dec 26 12:31:51.618062223 (501h2m51.489145258s) [upload=439732]: Waiting for error group managing segments... Dec 26 12:31:51.899859815 (501h2m51.77094285s) [noise=true , address=145.79.227.37:4566 , piece_id=BTRZJN3OJDJ5DJL62SG46SGPV4XPHIJTHAY26P7ZMTK7PQUZREIQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.904980976 (501h2m51.776064021s) [noise=true , address=145.79.227.4:4187 , piece_id=4QOJEQZ2KA3XK4HA2GB4XG3KSBN3CCFPRZH3ASAYTZVPRED3I3VQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.905001846 (501h2m51.776084882s) [noise=true , address=145.79.227.13:4802 , piece_id=OFPQKDI4CC6PP5JSCI66FW7WWN4M67R4KZWPSXA4X4SH442IUMJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.919874727 (501h2m51.790957772s) [noise=true , address=145.79.227.12:4765 , piece_id=77AV3VRATRZZKD6XF7K2FSJ3MCKV2PI3H4W22VVNF5MCN67CI5ZQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.921036388 (501h2m51.792119433s) [noise=true , address=145.79.227.47:4266 , piece_id=UE6ASC5I66WGD5EJ4FAI3VA22AIN4RZAFROVMOSOF7BW46SHCZ7Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.924479284 (501h2m51.795562329s) [noise=true , address=145.79.227.41:4086 , piece_id=JXGG6DIAYGDDXJOSSSXTCZOTJC2KXJJ3VBY5ZKJQUB5KXHQW6XVQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.926029998 (501h2m51.797113043s) [noise=true , address=145.79.227.42:4122 , piece_id=AS662XYX3O6ZUCR7JZSG7PJUGP5KVUSPTXXMR65BWUPGS7UED5NQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.927476752 (501h2m51.798559807s) [noise=true , address=145.79.227.49:4339 , piece_id=JLSGMCXHOM3NBBCSY3RR6CWWVQBWEXODRKHX4TSCKKGIQ5Q3GLJA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933634898 (501h2m51.804717933s) [noise=true , address=145.79.227.39:4015 , piece_id=XBKKBIJEHJNHAR5FTCBQP6C4TTPM5HHNOULDEI326OM4WS4AMFJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:51.933840370 (501h2m51.804923415s) [noise=true , address=145.79.227.17:4970 , piece_id=QYSAQSBFSSJ4TQRK2AITXJQEEORRUPWBQZXKL42IAOFLF2MTTIHA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.126336676 (501h2m51.997419721s) [noise=true , address=77.90.31.17:4773 , piece_id=AIOCFWA2SYRVTDXFH5H44ZTUZ2HDURSZ4CV5DAWADGIOMRPTZAEA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.267758879 (501h2m52.138841924s) [noise=true , address=45.91.116.9:4264 , piece_id=IKE5VKMKYHILFJ4BHPJQM366PXSZUPCLS6P5KZZIFTI4OR3X6FGA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858149345 (501h2m52.72923238s) [noise=true , address=45.91.116.8:4046 , piece_id=QP22YM4SYOYZS5GN625GZDGEGV5B74ZGWFHSFUKB7J5GCXZ2H62Q, seg_pos={0 0}, upload=439732]: Done uploading piece. err: <nil> Dec 26 12:31:52.858169524 (501h2m52.729252569s) [seg_pos={0 0}, upload=439732]: Segment reached optimal threshold of 13 pieces. Dec 26 12:31:52.858437404 (501h2m52.729520439s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858445700 (501h2m52.729528745s) [noise=true , address=45.91.116.12:4228 , piece_id=ZRCMS25DM4VVJBP6QDN5JS2A7CKIXYD6JWVR3F42HGL73DNG7BIA, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858575958 (501h2m52.729659003s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858687290 (501h2m52.729770335s) [noise=true , address=145.79.227.33:4870 , piece_id=MN2RZZFZ772C5MQYWP4R643H6FK5I5XGAM3BM3WNKC7TMNLQUYJQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858851242 (501h2m52.729934287s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:52.858860570 (501h2m52.729943615s) [noise=true , address=45.91.116.10:4084 , piece_id=U4FFAYJYA7UEOA3HFNB477ERWCMXY73YUOKS5BISTNT6RNDWXASQ, seg_pos={0 0}, upload=439732]: Long tail cancellation due to optimal threshold Dec 26 12:31:52.858882661 (501h2m52.729965706s) [seg_pos={0 0}, upload=439732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:31:52.858968074 (501h2m52.73005111s) [upload=439732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:31:52.858974116 (501h2m52.730057161s) [upload=439732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:31:53.152007553 (501h2m53.023090588s) [upload=439732]: Done uploading segments... Dec 26 12:31:53.152027951 (501h2m53.023110986s) [upload=439732]: Upload finished. err: <nil> Duration: 30.003586625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....XX] duration: 30.003751951s trace: https://jaeger.dev.storj.io/trace/4bd6684ae8bf6e94 FAIL (2025-12-26 12:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 12:31:15.231696911 (501h2m15.102779966s) [upload=439725]: Starting upload Dec 26 12:31:15.231781363 (501h2m15.102864418s) [upload=439725]: Done starting upload Dec 26 12:31:15.232059703 (501h2m15.103142758s) [upload=439725]: Uploading segments... Dec 26 12:31:15.232119797 (501h2m15.103202853s) [upload=439725]: Got next segment. Inline: false Dec 26 12:31:15.232151778 (501h2m15.103234834s) [upload=439725]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:31:15.232157139 (501h2m15.103240184s) [upload=439725]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:31:15.473517381 (501h2m15.344600416s) [seg_pos={0 0}, upload=439725]: Begin upload segment... Dec 26 12:31:15.473567055 (501h2m15.34465009s) [seg_pos={0 0}, upload=439725]: Done begin upload segment. Dec 26 12:31:15.473593756 (501h2m15.344676801s) [noise=true , address=145.79.227.11:4732 , piece_id=CCLU4JYEKLYZW5LGQSQEJVB6GPZDDU43MZ3NHQDFFNEIHEC7AXVA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.473770312 (501h2m15.344853357s) [noise=true , address=45.91.116.14:4191 , piece_id=4WUEVEGMDJ6MZLDCOPJM4TXZSXD67LKZEZ2HNN4CVI6HBKYKGNQA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.474430098 (501h2m15.345513143s) [noise=true , address=145.79.227.50:4374 , piece_id=6CCFOQ4GELVCNEALHHS4NZZQL4BKI6E3IVWSWF7KH3NO4NEM63JA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.474672039 (501h2m15.345755074s) [noise=true , address=77.90.31.14:4663 , piece_id=FNPZQGK47NDWLGC2DVXNF7BCQ4TJOXRPZLMIV2MJTALBLR3XWC3Q, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.475173103 (501h2m15.346256148s) [noise=true , address=145.79.227.38:4979 , piece_id=DKXWB243EATNBPYIQFOI2YKO7TETBLPOJQ75ZHKSPNOR6LWDM34A, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.475650652 (501h2m15.346733687s) [noise=true , address=145.79.227.32:4834 , piece_id=OBJSILI5UBUMBJOW5NCVZ3ADYXL2Y63JROMFE3K2GDSXNQHCDB6Q, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.476205679 (501h2m15.347288714s) [noise=true , address=145.79.227.47:4266 , piece_id=OVWXUJ2PMDHBX7T6S6T4LAUXKEKHTCTDIGNLU3CWXS7QQVW5VUXA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.476645396 (501h2m15.347728431s) [noise=true , address=45.91.116.10:4081 , piece_id=CKDR2FJIM7QVOYESGW2WE3BO2FLMZVJRSCTDRLVQUJPV5D2Z7DFA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.477093740 (501h2m15.348176785s) [noise=true , address=145.79.227.48:4303 , piece_id=2HSQX2J4TA43POPUIQTFKIT3LBZLORGKXFKE2ITAPXYBCO7L2GPA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.477634931 (501h2m15.348717976s) [noise=true , address=145.79.227.39:4014 , piece_id=7KKGKGPGOELVM7S3XRG4ZQT366CQ5WM5P6EC4EWPVLA4YY7DPUGA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.477882141 (501h2m15.348965177s) [noise=true , address=45.91.116.5:4297 , piece_id=LVEZMXD5DRARIFXXEKJE6PRTQBZNAT6LGBM4P3PT2Z6KV55S572A, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.478446486 (501h2m15.349529531s) [noise=true , address=145.79.227.37:4565 , piece_id=E4BUHLANV4C3PUFJYPH5ZRHYDWKNA4XXOZMQQWUXDDHJ5IC365AA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.479061367 (501h2m15.350144412s) [noise=true , address=145.79.227.5:4223 , piece_id=VA3PDH4EN5ZPFC7BK26NTQ4K2V3MHFTTUQAJNKTGKTH2HS4AN3MA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.479602136 (501h2m15.350685171s) [noise=true , address=45.91.116.9:4263 , piece_id=OWHC5EYSW3KGCMUY3VDFWCO7C2LE5GALJYAOL4ECG5X6ZAOQZQJA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.480255601 (501h2m15.351338636s) [noise=true , address=145.79.227.41:4087 , piece_id=7TCEZ3NKYVNQEWMAFDWIXHFQGHNDQ4ZWTFA3IGLZZZWA3QSGL2JQ, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:15.480866514 (501h2m15.35194955s) [noise=true , address=45.91.116.12:4225 , piece_id=SOJQXQFY4IQLHIMSFXNWIXUX3FIIP2TKIWQ32LNQVZMY5QBLH7QA, seg_pos={0 0}, upload=439725]: Uploading piece... Dec 26 12:31:17.183460381 (501h2m17.054543416s) [upload=439725]: Next returned nil segment Dec 26 12:31:17.183486581 (501h2m17.054569626s) [upload=439725]: Waiting for error group managing segments... Dec 26 12:31:17.466452377 (501h2m17.337535422s) [noise=true , address=145.79.227.48:4303 , piece_id=2HSQX2J4TA43POPUIQTFKIT3LBZLORGKXFKE2ITAPXYBCO7L2GPA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.476451156 (501h2m17.347534202s) [noise=true , address=145.79.227.5:4223 , piece_id=VA3PDH4EN5ZPFC7BK26NTQ4K2V3MHFTTUQAJNKTGKTH2HS4AN3MA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.481859915 (501h2m17.35294295s) [noise=true , address=145.79.227.11:4732 , piece_id=CCLU4JYEKLYZW5LGQSQEJVB6GPZDDU43MZ3NHQDFFNEIHEC7AXVA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.486533024 (501h2m17.35761607s) [noise=true , address=145.79.227.41:4087 , piece_id=7TCEZ3NKYVNQEWMAFDWIXHFQGHNDQ4ZWTFA3IGLZZZWA3QSGL2JQ, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.489654528 (501h2m17.360737573s) [noise=true , address=145.79.227.47:4266 , piece_id=OVWXUJ2PMDHBX7T6S6T4LAUXKEKHTCTDIGNLU3CWXS7QQVW5VUXA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.494365890 (501h2m17.365448925s) [noise=true , address=145.79.227.38:4979 , piece_id=DKXWB243EATNBPYIQFOI2YKO7TETBLPOJQ75ZHKSPNOR6LWDM34A, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.499674247 (501h2m17.370757292s) [noise=true , address=145.79.227.37:4565 , piece_id=E4BUHLANV4C3PUFJYPH5ZRHYDWKNA4XXOZMQQWUXDDHJ5IC365AA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.499801560 (501h2m17.370884605s) [noise=true , address=145.79.227.50:4374 , piece_id=6CCFOQ4GELVCNEALHHS4NZZQL4BKI6E3IVWSWF7KH3NO4NEM63JA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.505268580 (501h2m17.376351635s) [noise=true , address=145.79.227.32:4834 , piece_id=OBJSILI5UBUMBJOW5NCVZ3ADYXL2Y63JROMFE3K2GDSXNQHCDB6Q, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.518113048 (501h2m17.389196083s) [noise=true , address=145.79.227.39:4014 , piece_id=7KKGKGPGOELVM7S3XRG4ZQT366CQ5WM5P6EC4EWPVLA4YY7DPUGA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.723543253 (501h2m17.594626298s) [noise=true , address=45.91.116.14:4191 , piece_id=4WUEVEGMDJ6MZLDCOPJM4TXZSXD67LKZEZ2HNN4CVI6HBKYKGNQA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.827548347 (501h2m17.698631382s) [noise=true , address=45.91.116.5:4297 , piece_id=LVEZMXD5DRARIFXXEKJE6PRTQBZNAT6LGBM4P3PT2Z6KV55S572A, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.905279190 (501h2m17.776362225s) [noise=true , address=45.91.116.12:4225 , piece_id=SOJQXQFY4IQLHIMSFXNWIXUX3FIIP2TKIWQ32LNQVZMY5QBLH7QA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: <nil> Dec 26 12:31:17.905297616 (501h2m17.776380651s) [seg_pos={0 0}, upload=439725]: Segment reached optimal threshold of 13 pieces. Dec 26 12:31:17.905462459 (501h2m17.776545504s) [noise=true , address=45.91.116.10:4081 , piece_id=CKDR2FJIM7QVOYESGW2WE3BO2FLMZVJRSCTDRLVQUJPV5D2Z7DFA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:17.905470315 (501h2m17.77655335s) [noise=true , address=45.91.116.10:4081 , piece_id=CKDR2FJIM7QVOYESGW2WE3BO2FLMZVJRSCTDRLVQUJPV5D2Z7DFA, seg_pos={0 0}, upload=439725]: Long tail cancellation due to optimal threshold Dec 26 12:31:17.905492837 (501h2m17.776575872s) [noise=true , address=45.91.116.9:4263 , piece_id=OWHC5EYSW3KGCMUY3VDFWCO7C2LE5GALJYAOL4ECG5X6ZAOQZQJA, seg_pos={0 0}, upload=439725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:38302->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:38302->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38302->45.91.116.9:4263: use of closed network connection Dec 26 12:31:17.905513106 (501h2m17.776596141s) [noise=true , address=45.91.116.9:4263 , piece_id=OWHC5EYSW3KGCMUY3VDFWCO7C2LE5GALJYAOL4ECG5X6ZAOQZQJA, seg_pos={0 0}, upload=439725]: Long tail cancellation due to optimal threshold Dec 26 12:31:17.905576377 (501h2m17.776659412s) [noise=true , address=77.90.31.14:4663 , piece_id=FNPZQGK47NDWLGC2DVXNF7BCQ4TJOXRPZLMIV2MJTALBLR3XWC3Q, seg_pos={0 0}, upload=439725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 12:31:17.905581266 (501h2m17.776664311s) [noise=true , address=77.90.31.14:4663 , piece_id=FNPZQGK47NDWLGC2DVXNF7BCQ4TJOXRPZLMIV2MJTALBLR3XWC3Q, seg_pos={0 0}, upload=439725]: Long tail cancellation due to optimal threshold Dec 26 12:31:17.905596174 (501h2m17.776679219s) [seg_pos={0 0}, upload=439725]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:31:17.905679643 (501h2m17.776762678s) [upload=439725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:31:17.905686416 (501h2m17.776769461s) [upload=439725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:31:18.191953138 (501h2m18.063036183s) [upload=439725]: Done uploading segments... Dec 26 12:31:18.191975941 (501h2m18.063058987s) [upload=439725]: Upload finished. err: <nil> Duration: 30.00377807s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X............XX........X..X......X.......XXX...X.....X.....] duration: 4.433605801s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:31:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.433657859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 57.5480038s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:31:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 57.548017988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X............XX........X..X......X.......XXX...X.....X....] duration: 4.305707399s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:30:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.30573325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X.X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....X] duration: 30.003411471s trace: https://jaeger.dev.storj.io/trace/5e88826a3c07011e FAIL (2025-12-26 12:30:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: NNN5CJFUOWX2YLCNN2V46VQFN5UFGWF4QBN7UEB5EJGE6IJX3COQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: FJPAISRJEPTQQUZ3VIWRINTGAPG4HT3CNKVH7GBMVTXBYXYMAM2Q) context deadline exceeded ======LOGS====== h1m25.838050418s) [upload=439713]: Uploading segments... Dec 26 12:30:25.967213722 (501h1m25.838296767s) [upload=439713]: Got next segment. Inline: false Dec 26 12:30:25.967261083 (501h1m25.838344128s) [upload=439713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:30:25.967266774 (501h1m25.838349819s) [upload=439713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:30:26.210904512 (501h1m26.081987547s) [seg_pos={0 0}, upload=439713]: Begin upload segment... Dec 26 12:30:26.210975297 (501h1m26.082058372s) [seg_pos={0 0}, upload=439713]: Done begin upload segment. Dec 26 12:30:26.211022487 (501h1m26.082105532s) [noise=true , address=45.91.116.10:4081 , piece_id=ZMCRTULR2JLQPJTDSH22EEPGUSFZN6EGHI7SMOMR2ZCYD5HUBKLQ, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.211094114 (501h1m26.082177159s) [noise=true , address=145.79.227.47:4267 , piece_id=JNYOKBGOF5EFLBWWKOK2IQI4WT2OCRIOP2U7MDBFD2DML63J2WWA, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.211638770 (501h1m26.082721815s) [noise=true , address=77.90.31.18:4808 , piece_id=FJPAISRJEPTQQUZ3VIWRINTGAPG4HT3CNKVH7GBMVTXBYXYMAM2Q, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.212010288 (501h1m26.083093323s) [noise=true , address=145.79.227.39:4014 , piece_id=YWJYAOBOOMZE3PGINTB3HPVCDKYBCY4XJ5XOF4NI52AURDVMYHZQ, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.212371766 (501h1m26.083454811s) [noise=true , address=145.79.227.26:4654 , piece_id=573H4CSYLWTIK4HRPCEHYAK24C6FSQ36GL44YWATFLWZHSNXI5KQ, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.213181979 (501h1m26.084265024s) [noise=true , address=145.79.227.29:4941 , piece_id=I2LHJLL23BLI7CNOLDTTIFSAZKECPCLKOY354GU564MS4DRXBFFA, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.213329370 (501h1m26.084412405s) [noise=true , address=45.91.116.5:4300 , piece_id=ECUZOSYVYQ5KJPJ7YWVE6N2X3EO47J5QBVFYD7SAKZMJW76Y2ZMA, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.214579641 (501h1m26.085662686s) [noise=true , address=77.90.31.10:4518 , piece_id=NNN5CJFUOWX2YLCNN2V46VQFN5UFGWF4QBN7UEB5EJGE6IJX3COQ, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.215102015 (501h1m26.08618506s) [noise=true , address=145.79.227.45:4229 , piece_id=NAP2IYAABQ22UL4WZP6T7FYF6Z7D5KHN7WTTAPRANOUDXC46DDCQ, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.216043177 (501h1m26.087126232s) [noise=true , address=145.79.227.36:4519 , piece_id=LGXEET2DRA6NNUQPNWU4GFXPDR4LD2KU57QKUJXKXAMNTGGZGAXA, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.216782204 (501h1m26.087865239s) [noise=true , address=145.79.227.2:4114 , piece_id=JZO5YWOEZIJMCEITBLIOABV2QY2AHUUHEMG6U7OFPODKMM34CLUA, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.216939134 (501h1m26.088022169s) [noise=true , address=45.91.116.8:4046 , piece_id=LNSG3FY65AAMGAFSJONIJOBHP5NFMPD76LCQC2XX25WUKC5TT7PQ, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.218548117 (501h1m26.089631152s) [noise=true , address=145.79.227.11:4730 , piece_id=HL5M6BGLS52DCPD3XOI6DYQOG7XXDN2KNCK27KJF4AYFIL4YSNJQ, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.220232053 (501h1m26.091315108s) [noise=true , address=145.79.227.41:4085 , piece_id=UJNHQTXNMWRSNYEETZAVGBUTWWAVU6YKEBSPVU2TAU3HME5UHRNA, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.221012961 (501h1m26.092096006s) [noise=true , address=145.79.227.13:4803 , piece_id=QKTAGBKVQFEHZ3MRMEXMFKAJJ77DGOU5NGADCY5YFCOXKWY7GNLA, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:26.222386225 (501h1m26.09346926s) [noise=true , address=145.79.227.19:4077 , piece_id=3XVZDMUPNKTCPYWUQPGXCKLPB77XMSY2ROXNLB4CVU4WLRXJDB6Q, seg_pos={0 0}, upload=439713]: Uploading piece... Dec 26 12:30:27.920394631 (501h1m27.791477666s) [upload=439713]: Next returned nil segment Dec 26 12:30:27.920421582 (501h1m27.791504627s) [upload=439713]: Waiting for error group managing segments... Dec 26 12:30:28.208342845 (501h1m28.07942589s) [noise=true , address=145.79.227.45:4229 , piece_id=NAP2IYAABQ22UL4WZP6T7FYF6Z7D5KHN7WTTAPRANOUDXC46DDCQ, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.224837864 (501h1m28.095920909s) [noise=true , address=145.79.227.36:4519 , piece_id=LGXEET2DRA6NNUQPNWU4GFXPDR4LD2KU57QKUJXKXAMNTGGZGAXA, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.227245949 (501h1m28.098328994s) [noise=true , address=145.79.227.26:4654 , piece_id=573H4CSYLWTIK4HRPCEHYAK24C6FSQ36GL44YWATFLWZHSNXI5KQ, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.227805144 (501h1m28.098888189s) [noise=true , address=145.79.227.29:4941 , piece_id=I2LHJLL23BLI7CNOLDTTIFSAZKECPCLKOY354GU564MS4DRXBFFA, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.228013651 (501h1m28.099096686s) [noise=true , address=145.79.227.13:4803 , piece_id=QKTAGBKVQFEHZ3MRMEXMFKAJJ77DGOU5NGADCY5YFCOXKWY7GNLA, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.232686298 (501h1m28.103769343s) [noise=true , address=145.79.227.39:4014 , piece_id=YWJYAOBOOMZE3PGINTB3HPVCDKYBCY4XJ5XOF4NI52AURDVMYHZQ, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.240211208 (501h1m28.111294253s) [noise=true , address=145.79.227.47:4267 , piece_id=JNYOKBGOF5EFLBWWKOK2IQI4WT2OCRIOP2U7MDBFD2DML63J2WWA, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.240889599 (501h1m28.111972654s) [noise=true , address=145.79.227.19:4077 , piece_id=3XVZDMUPNKTCPYWUQPGXCKLPB77XMSY2ROXNLB4CVU4WLRXJDB6Q, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.244704484 (501h1m28.115787529s) [noise=true , address=145.79.227.41:4085 , piece_id=UJNHQTXNMWRSNYEETZAVGBUTWWAVU6YKEBSPVU2TAU3HME5UHRNA, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.247614184 (501h1m28.118697229s) [noise=true , address=145.79.227.2:4114 , piece_id=JZO5YWOEZIJMCEITBLIOABV2QY2AHUUHEMG6U7OFPODKMM34CLUA, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.260502245 (501h1m28.13158529s) [noise=true , address=145.79.227.11:4730 , piece_id=HL5M6BGLS52DCPD3XOI6DYQOG7XXDN2KNCK27KJF4AYFIL4YSNJQ, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.491513290 (501h1m28.362596335s) [noise=true , address=77.90.31.18:4808 , piece_id=FJPAISRJEPTQQUZ3VIWRINTGAPG4HT3CNKVH7GBMVTXBYXYMAM2Q, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.546565239 (501h1m28.417648274s) [noise=true , address=77.90.31.10:4518 , piece_id=NNN5CJFUOWX2YLCNN2V46VQFN5UFGWF4QBN7UEB5EJGE6IJX3COQ, seg_pos={0 0}, upload=439713]: Done uploading piece. err: <nil> Dec 26 12:30:28.546584535 (501h1m28.417667571s) [seg_pos={0 0}, upload=439713]: Segment reached optimal threshold of 13 pieces. Dec 26 12:30:28.546910827 (501h1m28.417993862s) [noise=true , address=45.91.116.8:4046 , piece_id=LNSG3FY65AAMGAFSJONIJOBHP5NFMPD76LCQC2XX25WUKC5TT7PQ, seg_pos={0 0}, upload=439713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 12:30:28.546924693 (501h1m28.418007769s) [noise=true , address=45.91.116.8:4046 , piece_id=LNSG3FY65AAMGAFSJONIJOBHP5NFMPD76LCQC2XX25WUKC5TT7PQ, seg_pos={0 0}, upload=439713]: Long tail cancellation due to optimal threshold Dec 26 12:30:28.547110397 (501h1m28.418193432s) [noise=true , address=45.91.116.10:4081 , piece_id=ZMCRTULR2JLQPJTDSH22EEPGUSFZN6EGHI7SMOMR2ZCYD5HUBKLQ, seg_pos={0 0}, upload=439713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:42950->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:42950->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42950->45.91.116.10:4081: use of closed network connection Dec 26 12:30:28.547117621 (501h1m28.418200666s) [noise=true , address=45.91.116.10:4081 , piece_id=ZMCRTULR2JLQPJTDSH22EEPGUSFZN6EGHI7SMOMR2ZCYD5HUBKLQ, seg_pos={0 0}, upload=439713]: Long tail cancellation due to optimal threshold Dec 26 12:30:28.547236367 (501h1m28.418319412s) [noise=true , address=45.91.116.5:4300 , piece_id=ECUZOSYVYQ5KJPJ7YWVE6N2X3EO47J5QBVFYD7SAKZMJW76Y2ZMA, seg_pos={0 0}, upload=439713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49012->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49012->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49012->45.91.116.5:4300: use of closed network connection Dec 26 12:30:28.547258810 (501h1m28.418341875s) [noise=true , address=45.91.116.5:4300 , piece_id=ECUZOSYVYQ5KJPJ7YWVE6N2X3EO47J5QBVFYD7SAKZMJW76Y2ZMA, seg_pos={0 0}, upload=439713]: Long tail cancellation due to optimal threshold Dec 26 12:30:28.547303064 (501h1m28.418386099s) [seg_pos={0 0}, upload=439713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:30:28.547413133 (501h1m28.418496169s) [upload=439713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:30:28.547419045 (501h1m28.41850208s) [upload=439713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:30:28.848079094 (501h1m28.719162129s) [upload=439713]: Done uploading segments... Dec 26 12:30:28.848099573 (501h1m28.719182608s) [upload=439713]: Upload finished. err: <nil> Duration: 30.003437069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 56.370192121s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:30:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 56.370218961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X............XX........X..X......X.......XXX...X.....X...] duration: 3.496718383s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:30:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496787714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....X............XX........X..X......X.......XXX...X.....X..] duration: 5.001543543s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:29:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001571707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X...X.X........X..XXXXX...X.X....X..X...X..XX.....XXXX.....] duration: 5.978265463s trace: https://jaeger.dev.storj.io/trace/163a73811a59f6c4 FAIL (2025-12-26 12:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: KYYSAH7UK45F7UMQXWP25BLE3ZCCMNZCREG76FOJUPBMTMG3V4XA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: QQSGYGZOCSTU2MZE4MK2ICMGUA5ZCTELYGKXTVAAQWT6MLMO23VA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: L2JEI2WMK62L7FEMAIRK4KXOIF7I52ZSC6B7TX66D7EPFWVVE6HA) context deadline exceeded ======LOGS====== Dec 26 12:25:05.299747126 (500h56m5.170830171s) [upload=439642]: Starting upload Dec 26 12:25:05.299859430 (500h56m5.170942475s) [upload=439642]: Done starting upload Dec 26 12:25:05.300075832 (500h56m5.171158877s) [upload=439642]: Uploading segments... Dec 26 12:25:05.300124585 (500h56m5.17120763s) [upload=439642]: Got next segment. Inline: false Dec 26 12:25:05.300172666 (500h56m5.171255701s) [upload=439642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:25:05.300177936 (500h56m5.171260981s) [upload=439642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:25:05.550554087 (500h56m5.421637132s) [seg_pos={0 0}, upload=439642]: Begin upload segment... Dec 26 12:25:05.550810024 (500h56m5.421893069s) [seg_pos={0 0}, upload=439642]: Done begin upload segment. Dec 26 12:25:05.550849118 (500h56m5.421932164s) [noise=true , address=145.79.227.19:4078 , piece_id=SVVYPCJ33NCWM2KPYMTWAHVVU34QVIQO77UH4MS46ZANDTO2UCJA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.550871652 (500h56m5.421954687s) [noise=true , address=45.91.116.10:4082 , piece_id=QQSGYGZOCSTU2MZE4MK2ICMGUA5ZCTELYGKXTVAAQWT6MLMO23VA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.551407412 (500h56m5.422490457s) [noise=true , address=145.79.227.16:4909 , piece_id=OXATAJBNT422IMWFFYGWTZCWR5L42MRI72FVWUABDV6ETZB6KATQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.551776795 (500h56m5.42285983s) [noise=true , address=145.79.227.38:4977 , piece_id=SBICE5BFAUOHTXI2BAOWWABFHPWZZPVFKBYQJPE4GCY35XODIHKA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.552135308 (500h56m5.423218353s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVHQP22DPEFSLFFHYHN5ILYLTPXXU4QENA4HINQ24J6GDBJKHA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.552474394 (500h56m5.423557439s) [noise=true , address=77.90.31.13:4627 , piece_id=L2JEI2WMK62L7FEMAIRK4KXOIF7I52ZSC6B7TX66D7EPFWVVE6HA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.552774915 (500h56m5.42385796s) [noise=true , address=145.79.227.52:4446 , piece_id=FBG53YAPEVI6IRGBGFGWH67DHA3DZWN74WV3ZXAZD6YHNN6CDWQQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.553077953 (500h56m5.424160998s) [noise=true , address=145.79.227.42:4123 , piece_id=2JXR4VZLCQOL3NKZOIT6XJEBWJASEWKQUMHN5WWHNTTMV5FOFGVA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.553588896 (500h56m5.424671941s) [noise=true , address=145.79.227.2:4114 , piece_id=VPWBOUMZIVDUHWGQYCK64THDEUSTW6774YH7ZHAXZCME4EIEL24A, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.554166635 (500h56m5.42524968s) [noise=true , address=145.79.227.6:4258 , piece_id=K3II2GLQXBBAUPKXJN2CN7XN35BJUB2ZOKSWBJCUVKLCH5BUPDTQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.554751960 (500h56m5.425835005s) [noise=true , address=45.91.116.14:4189 , piece_id=KYYSAH7UK45F7UMQXWP25BLE3ZCCMNZCREG76FOJUPBMTMG3V4XA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.554759574 (500h56m5.425842619s) [noise=true , address=77.90.31.21:4878 , piece_id=CHHS4MP3GMFXCIUGT5Q4BJC4XV5XLWIM4TGSWUCEKNVNBB5O2ITQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.555313369 (500h56m5.426396414s) [noise=true , address=45.91.116.12:4228 , piece_id=VGDL3DR2QTMYAAOD66LG5ODY5OG6LDEX7KTYV45WESXKYEDS5AWA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.556066473 (500h56m5.427149518s) [noise=true , address=145.79.227.41:4085 , piece_id=BYNCKHFEXPH6V64QD2LAS45SBRA7CZDSVBGFO2DDNM7RDQ435HLA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.556670183 (500h56m5.427753228s) [noise=true , address=77.90.31.19:4842 , piece_id=ZMVXDYNJ7HFQAUM3GSTP7JY7TDDNHXYODWJU5II2GE7VEZ7DIDLA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.557224679 (500h56m5.428307724s) [noise=true , address=45.91.116.8:4048 , piece_id=3R4AQUXXH53THTXZVXXJJTYYIPLT7VTN77SP6XEHRFMNRU2AKOVA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:07.273019852 (500h56m7.144102898s) [upload=439642]: Next returned nil segment Dec 26 12:25:07.273035321 (500h56m7.144118367s) [upload=439642]: Waiting for error group managing segments... Dec 26 12:25:07.557679544 (500h56m7.428762589s) [noise=true , address=145.79.227.41:4085 , piece_id=BYNCKHFEXPH6V64QD2LAS45SBRA7CZDSVBGFO2DDNM7RDQ435HLA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.559697667 (500h56m7.430780712s) [noise=true , address=145.79.227.19:4078 , piece_id=SVVYPCJ33NCWM2KPYMTWAHVVU34QVIQO77UH4MS46ZANDTO2UCJA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.560670810 (500h56m7.431753855s) [noise=true , address=145.79.227.42:4123 , piece_id=2JXR4VZLCQOL3NKZOIT6XJEBWJASEWKQUMHN5WWHNTTMV5FOFGVA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.561929176 (500h56m7.433012211s) [noise=true , address=145.79.227.6:4258 , piece_id=K3II2GLQXBBAUPKXJN2CN7XN35BJUB2ZOKSWBJCUVKLCH5BUPDTQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.562762613 (500h56m7.433845648s) [noise=true , address=145.79.227.38:4977 , piece_id=SBICE5BFAUOHTXI2BAOWWABFHPWZZPVFKBYQJPE4GCY35XODIHKA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.563173847 (500h56m7.434256892s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVHQP22DPEFSLFFHYHN5ILYLTPXXU4QENA4HINQ24J6GDBJKHA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.586836213 (500h56m7.457919258s) [noise=true , address=145.79.227.16:4909 , piece_id=OXATAJBNT422IMWFFYGWTZCWR5L42MRI72FVWUABDV6ETZB6KATQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.586880568 (500h56m7.457963603s) [noise=true , address=145.79.227.2:4114 , piece_id=VPWBOUMZIVDUHWGQYCK64THDEUSTW6774YH7ZHAXZCME4EIEL24A, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.589914806 (500h56m7.460997851s) [noise=true , address=145.79.227.52:4446 , piece_id=FBG53YAPEVI6IRGBGFGWH67DHA3DZWN74WV3ZXAZD6YHNN6CDWQQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.809328317 (500h56m7.680411362s) [noise=true , address=77.90.31.21:4878 , piece_id=CHHS4MP3GMFXCIUGT5Q4BJC4XV5XLWIM4TGSWUCEKNVNBB5O2ITQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.873955869 (500h56m7.745038904s) [noise=true , address=45.91.116.14:4189 , piece_id=KYYSAH7UK45F7UMQXWP25BLE3ZCCMNZCREG76FOJUPBMTMG3V4XA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.980426308 (500h56m7.851509363s) [noise=true , address=45.91.116.10:4082 , piece_id=QQSGYGZOCSTU2MZE4MK2ICMGUA5ZCTELYGKXTVAAQWT6MLMO23VA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:08.151633726 (500h56m8.022716761s) [noise=true , address=77.90.31.13:4627 , piece_id=L2JEI2WMK62L7FEMAIRK4KXOIF7I52ZSC6B7TX66D7EPFWVVE6HA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:08.151660247 (500h56m8.022743292s) [seg_pos={0 0}, upload=439642]: Segment reached optimal threshold of 13 pieces. Dec 26 12:25:08.151909632 (500h56m8.022992677s) [noise=true , address=45.91.116.12:4228 , piece_id=VGDL3DR2QTMYAAOD66LG5ODY5OG6LDEX7KTYV45WESXKYEDS5AWA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:25:08.151919301 (500h56m8.023002346s) [noise=true , address=45.91.116.12:4228 , piece_id=VGDL3DR2QTMYAAOD66LG5ODY5OG6LDEX7KTYV45WESXKYEDS5AWA, seg_pos={0 0}, upload=439642]: Long tail cancellation due to optimal threshold Dec 26 12:25:08.152190877 (500h56m8.023273922s) [noise=true , address=77.90.31.19:4842 , piece_id=ZMVXDYNJ7HFQAUM3GSTP7JY7TDDNHXYODWJU5II2GE7VEZ7DIDLA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 12:25:08.152204975 (500h56m8.02328802s) [noise=true , address=77.90.31.19:4842 , piece_id=ZMVXDYNJ7HFQAUM3GSTP7JY7TDDNHXYODWJU5II2GE7VEZ7DIDLA, seg_pos={0 0}, upload=439642]: Long tail cancellation due to optimal threshold Dec 26 12:25:08.152216977 (500h56m8.023300022s) [noise=true , address=45.91.116.8:4048 , piece_id=3R4AQUXXH53THTXZVXXJJTYYIPLT7VTN77SP6XEHRFMNRU2AKOVA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:51122->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:25:08.152228258 (500h56m8.023311303s) [noise=true , address=45.91.116.8:4048 , piece_id=3R4AQUXXH53THTXZVXXJJTYYIPLT7VTN77SP6XEHRFMNRU2AKOVA, seg_pos={0 0}, upload=439642]: Long tail cancellation due to optimal threshold Dec 26 12:25:08.152247105 (500h56m8.02333015s) [seg_pos={0 0}, upload=439642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:25:08.152343889 (500h56m8.023426924s) [upload=439642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:25:08.152351383 (500h56m8.023434428s) [upload=439642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:25:08.438806374 (500h56m8.309889419s) [upload=439642]: Done uploading segments... Dec 26 12:25:08.438830559 (500h56m8.309913604s) [upload=439642]: Upload finished. err: <nil> Duration: 5.978293656s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X....X............XX........X..X......X.......XXX...X.....X.] duration: 3.368024738s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:29:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.368054664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X...X.X........X..XXXXX...X.X....X..X...X..XX.....XXXX....] duration: 6.147839041s trace: https://jaeger.dev.storj.io/trace/163a73811a59f6c4 FAIL (2025-12-26 12:29:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: KYYSAH7UK45F7UMQXWP25BLE3ZCCMNZCREG76FOJUPBMTMG3V4XA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: QQSGYGZOCSTU2MZE4MK2ICMGUA5ZCTELYGKXTVAAQWT6MLMO23VA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: L2JEI2WMK62L7FEMAIRK4KXOIF7I52ZSC6B7TX66D7EPFWVVE6HA) context deadline exceeded ======LOGS====== Dec 26 12:25:05.299747126 (500h56m5.170830171s) [upload=439642]: Starting upload Dec 26 12:25:05.299859430 (500h56m5.170942475s) [upload=439642]: Done starting upload Dec 26 12:25:05.300075832 (500h56m5.171158877s) [upload=439642]: Uploading segments... Dec 26 12:25:05.300124585 (500h56m5.17120763s) [upload=439642]: Got next segment. Inline: false Dec 26 12:25:05.300172666 (500h56m5.171255701s) [upload=439642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:25:05.300177936 (500h56m5.171260981s) [upload=439642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:25:05.550554087 (500h56m5.421637132s) [seg_pos={0 0}, upload=439642]: Begin upload segment... Dec 26 12:25:05.550810024 (500h56m5.421893069s) [seg_pos={0 0}, upload=439642]: Done begin upload segment. Dec 26 12:25:05.550849118 (500h56m5.421932164s) [noise=true , address=145.79.227.19:4078 , piece_id=SVVYPCJ33NCWM2KPYMTWAHVVU34QVIQO77UH4MS46ZANDTO2UCJA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.550871652 (500h56m5.421954687s) [noise=true , address=45.91.116.10:4082 , piece_id=QQSGYGZOCSTU2MZE4MK2ICMGUA5ZCTELYGKXTVAAQWT6MLMO23VA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.551407412 (500h56m5.422490457s) [noise=true , address=145.79.227.16:4909 , piece_id=OXATAJBNT422IMWFFYGWTZCWR5L42MRI72FVWUABDV6ETZB6KATQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.551776795 (500h56m5.42285983s) [noise=true , address=145.79.227.38:4977 , piece_id=SBICE5BFAUOHTXI2BAOWWABFHPWZZPVFKBYQJPE4GCY35XODIHKA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.552135308 (500h56m5.423218353s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVHQP22DPEFSLFFHYHN5ILYLTPXXU4QENA4HINQ24J6GDBJKHA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.552474394 (500h56m5.423557439s) [noise=true , address=77.90.31.13:4627 , piece_id=L2JEI2WMK62L7FEMAIRK4KXOIF7I52ZSC6B7TX66D7EPFWVVE6HA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.552774915 (500h56m5.42385796s) [noise=true , address=145.79.227.52:4446 , piece_id=FBG53YAPEVI6IRGBGFGWH67DHA3DZWN74WV3ZXAZD6YHNN6CDWQQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.553077953 (500h56m5.424160998s) [noise=true , address=145.79.227.42:4123 , piece_id=2JXR4VZLCQOL3NKZOIT6XJEBWJASEWKQUMHN5WWHNTTMV5FOFGVA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.553588896 (500h56m5.424671941s) [noise=true , address=145.79.227.2:4114 , piece_id=VPWBOUMZIVDUHWGQYCK64THDEUSTW6774YH7ZHAXZCME4EIEL24A, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.554166635 (500h56m5.42524968s) [noise=true , address=145.79.227.6:4258 , piece_id=K3II2GLQXBBAUPKXJN2CN7XN35BJUB2ZOKSWBJCUVKLCH5BUPDTQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.554751960 (500h56m5.425835005s) [noise=true , address=45.91.116.14:4189 , piece_id=KYYSAH7UK45F7UMQXWP25BLE3ZCCMNZCREG76FOJUPBMTMG3V4XA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.554759574 (500h56m5.425842619s) [noise=true , address=77.90.31.21:4878 , piece_id=CHHS4MP3GMFXCIUGT5Q4BJC4XV5XLWIM4TGSWUCEKNVNBB5O2ITQ, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.555313369 (500h56m5.426396414s) [noise=true , address=45.91.116.12:4228 , piece_id=VGDL3DR2QTMYAAOD66LG5ODY5OG6LDEX7KTYV45WESXKYEDS5AWA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.556066473 (500h56m5.427149518s) [noise=true , address=145.79.227.41:4085 , piece_id=BYNCKHFEXPH6V64QD2LAS45SBRA7CZDSVBGFO2DDNM7RDQ435HLA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.556670183 (500h56m5.427753228s) [noise=true , address=77.90.31.19:4842 , piece_id=ZMVXDYNJ7HFQAUM3GSTP7JY7TDDNHXYODWJU5II2GE7VEZ7DIDLA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:05.557224679 (500h56m5.428307724s) [noise=true , address=45.91.116.8:4048 , piece_id=3R4AQUXXH53THTXZVXXJJTYYIPLT7VTN77SP6XEHRFMNRU2AKOVA, seg_pos={0 0}, upload=439642]: Uploading piece... Dec 26 12:25:07.273019852 (500h56m7.144102898s) [upload=439642]: Next returned nil segment Dec 26 12:25:07.273035321 (500h56m7.144118367s) [upload=439642]: Waiting for error group managing segments... Dec 26 12:25:07.557679544 (500h56m7.428762589s) [noise=true , address=145.79.227.41:4085 , piece_id=BYNCKHFEXPH6V64QD2LAS45SBRA7CZDSVBGFO2DDNM7RDQ435HLA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.559697667 (500h56m7.430780712s) [noise=true , address=145.79.227.19:4078 , piece_id=SVVYPCJ33NCWM2KPYMTWAHVVU34QVIQO77UH4MS46ZANDTO2UCJA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.560670810 (500h56m7.431753855s) [noise=true , address=145.79.227.42:4123 , piece_id=2JXR4VZLCQOL3NKZOIT6XJEBWJASEWKQUMHN5WWHNTTMV5FOFGVA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.561929176 (500h56m7.433012211s) [noise=true , address=145.79.227.6:4258 , piece_id=K3II2GLQXBBAUPKXJN2CN7XN35BJUB2ZOKSWBJCUVKLCH5BUPDTQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.562762613 (500h56m7.433845648s) [noise=true , address=145.79.227.38:4977 , piece_id=SBICE5BFAUOHTXI2BAOWWABFHPWZZPVFKBYQJPE4GCY35XODIHKA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.563173847 (500h56m7.434256892s) [noise=true , address=145.79.227.36:4518 , piece_id=FKBVHQP22DPEFSLFFHYHN5ILYLTPXXU4QENA4HINQ24J6GDBJKHA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.586836213 (500h56m7.457919258s) [noise=true , address=145.79.227.16:4909 , piece_id=OXATAJBNT422IMWFFYGWTZCWR5L42MRI72FVWUABDV6ETZB6KATQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.586880568 (500h56m7.457963603s) [noise=true , address=145.79.227.2:4114 , piece_id=VPWBOUMZIVDUHWGQYCK64THDEUSTW6774YH7ZHAXZCME4EIEL24A, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.589914806 (500h56m7.460997851s) [noise=true , address=145.79.227.52:4446 , piece_id=FBG53YAPEVI6IRGBGFGWH67DHA3DZWN74WV3ZXAZD6YHNN6CDWQQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.809328317 (500h56m7.680411362s) [noise=true , address=77.90.31.21:4878 , piece_id=CHHS4MP3GMFXCIUGT5Q4BJC4XV5XLWIM4TGSWUCEKNVNBB5O2ITQ, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.873955869 (500h56m7.745038904s) [noise=true , address=45.91.116.14:4189 , piece_id=KYYSAH7UK45F7UMQXWP25BLE3ZCCMNZCREG76FOJUPBMTMG3V4XA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:07.980426308 (500h56m7.851509363s) [noise=true , address=45.91.116.10:4082 , piece_id=QQSGYGZOCSTU2MZE4MK2ICMGUA5ZCTELYGKXTVAAQWT6MLMO23VA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:08.151633726 (500h56m8.022716761s) [noise=true , address=77.90.31.13:4627 , piece_id=L2JEI2WMK62L7FEMAIRK4KXOIF7I52ZSC6B7TX66D7EPFWVVE6HA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: <nil> Dec 26 12:25:08.151660247 (500h56m8.022743292s) [seg_pos={0 0}, upload=439642]: Segment reached optimal threshold of 13 pieces. Dec 26 12:25:08.151909632 (500h56m8.022992677s) [noise=true , address=45.91.116.12:4228 , piece_id=VGDL3DR2QTMYAAOD66LG5ODY5OG6LDEX7KTYV45WESXKYEDS5AWA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 12:25:08.151919301 (500h56m8.023002346s) [noise=true , address=45.91.116.12:4228 , piece_id=VGDL3DR2QTMYAAOD66LG5ODY5OG6LDEX7KTYV45WESXKYEDS5AWA, seg_pos={0 0}, upload=439642]: Long tail cancellation due to optimal threshold Dec 26 12:25:08.152190877 (500h56m8.023273922s) [noise=true , address=77.90.31.19:4842 , piece_id=ZMVXDYNJ7HFQAUM3GSTP7JY7TDDNHXYODWJU5II2GE7VEZ7DIDLA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 12:25:08.152204975 (500h56m8.02328802s) [noise=true , address=77.90.31.19:4842 , piece_id=ZMVXDYNJ7HFQAUM3GSTP7JY7TDDNHXYODWJU5II2GE7VEZ7DIDLA, seg_pos={0 0}, upload=439642]: Long tail cancellation due to optimal threshold Dec 26 12:25:08.152216977 (500h56m8.023300022s) [noise=true , address=45.91.116.8:4048 , piece_id=3R4AQUXXH53THTXZVXXJJTYYIPLT7VTN77SP6XEHRFMNRU2AKOVA, seg_pos={0 0}, upload=439642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:51122->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:25:08.152228258 (500h56m8.023311303s) [noise=true , address=45.91.116.8:4048 , piece_id=3R4AQUXXH53THTXZVXXJJTYYIPLT7VTN77SP6XEHRFMNRU2AKOVA, seg_pos={0 0}, upload=439642]: Long tail cancellation due to optimal threshold Dec 26 12:25:08.152247105 (500h56m8.02333015s) [seg_pos={0 0}, upload=439642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:25:08.152343889 (500h56m8.023426924s) [upload=439642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:25:08.152351383 (500h56m8.023434428s) [upload=439642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:25:08.438806374 (500h56m8.309889419s) [upload=439642]: Done uploading segments... Dec 26 12:25:08.438830559 (500h56m8.309913604s) [upload=439642]: Upload finished. err: <nil> Duration: 6.147883264s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X....X............XX........X..X......X.......XXX...X.....X] duration: 5.001107604s trace: https://jaeger.dev.storj.io/trace/7ff5ca2b29d4eca2 FAIL (2025-12-26 12:29:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001132511s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)