Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [........X............X......X.....X..X....X.........X..X...X] duration:
1m0.000750913s trace:
https://jaeger.dev.storj.io/trace/7f516ba4e20a24a4
FAIL (2026-05-16 20:01:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000761285s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
15.96735411s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:01:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 15.967378156s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [..XX.X..X.X....X.........X.........................X........] duration:
4.504508192s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 20:00:50 UTC)
orbiter warning: context deadline exceeded
Duration: 4.504532418s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.113858388s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:00:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 5.113894497s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.........X............X......X.....X..X....X.........X..X...] duration:
50.883895618s trace:
https://jaeger.dev.storj.io/trace/212429f77169093f
FAIL (2026-05-16 20:00:12 UTC)
orbiter warning: context deadline exceeded
Duration: 50.883912531s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [...XX.X..X.X....X.........X.........................X.......] duration:
4.465298678s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 20:00:11 UTC)
orbiter warning: context deadline exceeded
Duration: 4.465325789s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
10.19293068s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:59:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 10.192967169s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..........X............X......X.....X..X....X.........X..X..] duration:
13.104063598s trace:
https://jaeger.dev.storj.io/trace/212429f77169093f
FAIL (2026-05-16 19:59:19 UTC)
orbiter warning: context deadline exceeded
Duration: 13.104107331s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
4.252214524s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:59:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.25223861s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [....XX.X..X.X....X.........X.........................X......] duration:
4.953881465s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 19:59:09 UTC)
orbiter warning: context deadline exceeded
Duration: 4.953926921s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
12.615994995s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:59:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 12.616016909s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...........X............X......X.....X..X....X.........X..X.] duration:
53.765338792s trace:
https://jaeger.dev.storj.io/trace/212429f77169093f
FAIL (2026-05-16 19:58:55 UTC)
orbiter warning: context deadline exceeded
Duration: 53.765351416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.....XX.X..X.X....X.........X.........................X.....] duration:
4.18013888s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 19:58:31 UTC)
orbiter warning: context deadline exceeded
Duration: 4.1801548s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
14.009744937s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:58:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 14.00976814s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
4.242127836s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:58:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.24215566s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [............X............X......X.....X..X....X.........X..X] duration:
1m0.000750738s trace:
https://jaeger.dev.storj.io/trace/212429f77169093f
FAIL (2026-05-16 19:58:01 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000768693s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [......XX.X..X.X....X.........X.........................X....] duration:
4.942476281s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 19:57:36 UTC)
orbiter warning: context deadline exceeded
Duration: 4.94249016s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
18.272985533s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:57:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 18.273019458s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
4.174350673s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:57:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.17437505s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
4.21396484s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:56:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.213993623s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [.......XX.X..X.X....X.........X.........................X...] duration:
4.5271206s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 19:56:27 UTC)
orbiter warning: context deadline exceeded
Duration: 4.527149535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
8.84008861s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:56:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.840115059s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.............X............X......X.....X..X....X.........X..] duration:
7.923520032s trace:
https://jaeger.dev.storj.io/trace/16380bcd7d539d15
FAIL (2026-05-16 19:55:55 UTC)
orbiter warning: context deadline exceeded
Duration: 7.923533096s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
4.219196237s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:55:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.219221524s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [X.......XX.X..X.X....X.........X.........................X..] duration:
4.560711617s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 19:55:24 UTC)
orbiter warning: context deadline exceeded
Duration: 4.560736625s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [X.............X............X......X.....X..X....X.........X.] duration:
30.516314055s trace:
https://jaeger.dev.storj.io/trace/16380bcd7d539d15
FAIL (2026-05-16 19:55:24 UTC)
orbiter warning: context deadline exceeded
Duration: 30.516344222s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
7.047328545s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:55:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 7.047350407s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
4.199075826s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:54:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.199098679s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [.X.......XX.X..X.X....X.........X.........................X.] duration:
4.774822421s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 19:54:58 UTC)
orbiter warning: context deadline exceeded
Duration: 4.774848491s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.X.............X............X......X.....X..X....X.........X] duration:
1m0.001149528s trace:
https://jaeger.dev.storj.io/trace/16380bcd7d539d15
FAIL (2026-05-16 19:54:54 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001170589s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
7.883824994s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:54:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 7.883874409s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
4.618756009s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:54:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.618777929s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [..X.......XX.X..X.X....X.........X.........................X] duration:
5.001066757s trace:
https://jaeger.dev.storj.io/trace/5e7f227157705318
FAIL (2026-05-16 19:54:02 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001080362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [..X.............X............X......X.....X..X....X.........] duration:
52.252459539s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:53:49 UTC)
orbiter warning: context deadline exceeded
Duration: 52.252470319s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
4.592224178s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:53:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.592248897s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
8.40103062s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:53:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.401044727s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [X..X.......XX.X..X.X....X.........X.........................] duration:
4.500710151s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:53:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.500732815s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
11.349602348s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:53:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 11.349625641s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...X.............X............X......X.....X..X....X........] duration:
50.547045464s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:52:56 UTC)
orbiter warning: context deadline exceeded
Duration: 50.547097193s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
4.23920231s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:52:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.239225213s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [XX..X.......XX.X..X.X....X.........X........................] duration:
4.474784321s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:52:43 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.474805662s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [.XX..X.......XX.X..X.X....X.........X.......................] duration:
5.000884204s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:52:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000906876s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
4.784020732s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:52:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.784047843s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
6.090568596s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:51:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.090594014s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [....X.............X............X......X.....X..X....X.......] duration:
9.954729604s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:51:18 UTC)
orbiter warning: context deadline exceeded
Duration: 9.954745967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
8.701737629s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:51:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.701760491s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [..XX..X.......XX.X..X.X....X.........X......................] duration:
4.713919144s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:51:11 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.713938832s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
4.640838009s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:51:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.64085501s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [X....X.............X............X......X.....X..X....X......] duration:
22.134842862s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:50:51 UTC)
orbiter warning: context deadline exceeded
Duration: 22.134853682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [...XX..X.......XX.X..X.X....X.........X.....................] duration:
4.754758415s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:50:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.754785187s
core
uplink
us-select-1
us1-placement-20
|