Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X.........................X..........X.....X...............] duration:
3.793513515s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:17:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.793552489s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
11.770692389s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:17:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 11.770726243s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X.........................X..........X.....X..............] duration:
4.508826186s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:17:33 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.508840994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
4.652741099s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:17:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.652766096s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X.........................X..........X.....X.............] duration:
4.4299635s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:17:14 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.429992455s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
13.09391519s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:16:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 13.093939587s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
4.658106387s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:16:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.658132256s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....X.........................X..........X.....X............] duration:
5.001138719s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:16:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001160853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [....X......X.....X..X....X.........X..X...X.......X..X......] duration:
14.756501782s trace:
https://jaeger.dev.storj.io/trace/402d7bdace7deb49
FAIL (2026-05-16 20:16:33 UTC)
orbiter warning: context deadline exceeded
Duration: 14.756525065s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
11.972224998s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:16:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 11.972254465s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....X.........................X..........X.....X...........] duration:
3.91259102s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:16:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.912609406s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
4.59500847s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:16:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.595035351s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [.....X......X.....X..X....X.........X..X...X.......X..X.....] duration:
34.540791797s trace:
https://jaeger.dev.storj.io/trace/402d7bdace7deb49
FAIL (2026-05-16 20:16:00 UTC)
orbiter warning: context deadline exceeded
Duration: 34.540815863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......X.........................X..........X.....X..........] duration:
3.980433411s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:15:43 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.980456966s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
8.317805935s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:15:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.317822195s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
4.192213681s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:15:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.192241223s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......X.........................X..........X.....X.........] duration:
4.023668603s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:15:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.023693038s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
6.309744606s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:14:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.309765125s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
4.658694088s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:14:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.658725288s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [......X......X.....X..X....X.........X..X...X.......X..X....] duration:
10.067963438s trace:
https://jaeger.dev.storj.io/trace/402d7bdace7deb49
FAIL (2026-05-16 20:14:29 UTC)
orbiter warning: context deadline exceeded
Duration: 10.067987032s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.107485997s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:14:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.107509211s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........X.........................X..........X.....X........] duration:
4.10592315s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:14:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.105946784s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........X.........................X..........X.....X.......] duration:
4.90219867s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:13:46 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.9022201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
10.430184701s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:13:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 10.430210961s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [X.........X.........................X..........X.....X......] duration:
4.17093215s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:13:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.170956676s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
4.306369417s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:13:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.306389985s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [.......X......X.....X..X....X.........X..X...X.......X..X...] duration:
8.023351363s trace:
https://jaeger.dev.storj.io/trace/402d7bdace7deb49
FAIL (2026-05-16 20:13:18 UTC)
orbiter warning: context deadline exceeded
Duration: 8.023371312s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [........X......X.....X..X....X.........X..X...X.......X..X..] duration:
48.261101462s trace:
https://jaeger.dev.storj.io/trace/402d7bdace7deb49
FAIL (2026-05-16 20:13:02 UTC)
orbiter warning: context deadline exceeded
Duration: 48.261128753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
7.108241365s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:12:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 7.10827569s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.X.........X.........................X..........X.....X.....] duration:
4.010230258s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:12:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.010256778s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.109986813s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:12:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 5.110010167s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..X.........X.........................X..........X.....X....] duration:
5.000568928s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:11:58 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000593154s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
11.028586699s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:11:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 11.028609662s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [.........X......X.....X..X....X.........X..X...X.......X..X.] duration:
7.76717515s trace:
https://jaeger.dev.storj.io/trace/402d7bdace7deb49
FAIL (2026-05-16 20:11:20 UTC)
orbiter warning: context deadline exceeded
Duration: 7.767190218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [..........X......X.....X..X....X.........X..X...X.......X..X] duration:
1m0.001249215s trace:
https://jaeger.dev.storj.io/trace/402d7bdace7deb49
FAIL (2026-05-16 20:11:12 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.00125745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
4.724384588s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:11:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.724409827s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...X.........X.........................X..........X.....X...] duration:
4.066784715s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:10:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.066823679s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
8.215532637s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:10:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.21555984s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
4.589661482s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:10:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.589682931s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [...........X......X.....X..X....X.........X..X...X.......X..] duration:
49.706808372s trace:
https://jaeger.dev.storj.io/trace/2cd97297f61b58fd
FAIL (2026-05-16 20:10:12 UTC)
orbiter warning: context deadline exceeded
Duration: 49.706825826s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
4.601056553s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:09:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.60107632s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....X.........X.........................X..........X.....X..] duration:
4.072673973s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:09:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.072801667s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
7.780807101s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:09:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 7.780828351s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
4.318141589s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:09:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.318167067s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [............X......X.....X..X....X.........X..X...X.......X.] duration:
33.207003698s trace:
https://jaeger.dev.storj.io/trace/2cd97297f61b58fd
FAIL (2026-05-16 20:09:11 UTC)
orbiter warning: context deadline exceeded
Duration: 33.207016302s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
4.659423642s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 20:09:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.659449391s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [X....X.........X.........................X..........X.....X.] duration:
3.893092289s trace:
https://jaeger.dev.storj.io/trace/799b0430a3589c90
FAIL (2026-05-16 20:09:00 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.893127455s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.03265579s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:08:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.032683213s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
8.422471963s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 20:08:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.422496679s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [X............X......X.....X..X....X.........X..X...X.......X] duration:
1m0.000663807s trace:
https://jaeger.dev.storj.io/trace/2cd97297f61b58fd
FAIL (2026-05-16 20:08:38 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000682103s
core
uplink
us-select-1
us1-placement-20
|