Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
4.640838009s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:51:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.64085501s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [X....X.............X............X......X.....X..X....X......] duration:
22.134842862s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:50:51 UTC)
orbiter warning: context deadline exceeded
Duration: 22.134853682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [...XX..X.......XX.X..X.X....X.........X.....................] duration:
4.754758415s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:50:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.754785187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
4.230951285s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:50:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.230977466s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.X....X.............X............X......X.....X..X....X.....] duration:
56.416115917s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:50:28 UTC)
orbiter warning: context deadline exceeded
Duration: 56.41612713s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
6.907382745s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:50:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.907410298s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
11 out of 60 is failed (threshold=6) [X...XX..X.......XX.X..X.X....X.........X....................] duration:
3.956200642s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:50:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.956223344s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
4.983857578s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:49:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.983880623s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..X....X.............X............X......X.....X..X....X....] duration:
52.608279356s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:49:32 UTC)
orbiter warning: context deadline exceeded
Duration: 52.608305927s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.669163574s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:49:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.66919333s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
11 out of 60 is failed (threshold=6) [.X...XX..X.......XX.X..X.X....X.........X...................] duration:
3.876303999s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:49:14 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.876329889s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
11 out of 60 is failed (threshold=6) [..X...XX..X.......XX.X..X.X....X.........X..................] duration:
4.270442354s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:48:58 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.270462802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
4.274174025s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:48:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.274200256s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
6.154005446s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:48:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.154030965s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.052241451s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:48:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.052286928s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
4.613487059s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:47:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.613516905s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
11 out of 60 is failed (threshold=6) [...X...XX..X.......XX.X..X.X....X.........X.................] duration:
3.788570845s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:47:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.788594951s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...X....X.............X............X......X.....X..X....X...] duration:
9.3526787s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:47:51 UTC)
orbiter warning: context deadline exceeded
Duration: 9.352696264s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.198965773s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:47:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 6.198987244s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [....X....X.............X............X......X.....X..X....X..] duration:
6.635414735s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:47:24 UTC)
orbiter warning: context deadline exceeded
Duration: 6.635428721s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.60111359s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:47:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 6.601138517s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [X....X....X.............X............X......X.....X..X....X.] duration:
50.143392227s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:47:17 UTC)
orbiter warning: context deadline exceeded
Duration: 50.143767674s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
11 out of 60 is failed (threshold=6) [....X...XX..X.......XX.X..X.X....X.........X................] duration:
3.984800294s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:46:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.98482915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
10.960597996s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:46:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 10.960618136s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
4.206190075s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:46:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.206216316s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.X....X....X.............X............X......X.....X..X....X] duration:
1m0.000669703s trace:
https://jaeger.dev.storj.io/trace/5bdea4c7068d790f
FAIL (2026-05-16 19:46:26 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000684653s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
9.507616259s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:46:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 9.507640145s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.498993784s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:45:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 6.499020154s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [X....X...XX..X.......XX.X..X.X....X.........X...............] duration:
3.854690266s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:45:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.854719792s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [XX....X...XX..X.......XX.X..X.X....X.........X..............] duration:
3.800287031s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:45:30 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.800310836s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
7.165687377s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:45:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 7.165710572s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..X....X....X.............X............X......X.....X..X....] duration:
38.453921504s trace:
https://jaeger.dev.storj.io/trace/6b00f0e5b917eb2f
FAIL (2026-05-16 19:45:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 38.453946372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.XX....X...XX..X.......XX.X..X.X....X.........X.............] duration:
4.012311652s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:45:09 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.012334905s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
4.66062529s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:44:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.660645309s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [..XX....X...XX..X.......XX.X..X.X....X.........X............] duration:
3.938248093s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:44:20 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.938273391s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
8.143544309s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:44:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.143572733s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...X....X....X.............X............X......X.....X..X...] duration:
9.233358661s trace:
https://jaeger.dev.storj.io/trace/6b00f0e5b917eb2f
FAIL (2026-05-16 19:44:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.233376124s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
4.279322507s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:43:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.27934489s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [X..XX....X...XX..X.......XX.X..X.X....X.........X...........] duration:
3.849668954s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:43:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.849695423s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [....X....X....X.............X............X......X.....X..X..] duration:
18.533599648s trace:
https://jaeger.dev.storj.io/trace/6b00f0e5b917eb2f
FAIL (2026-05-16 19:43:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.533615618s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.....X....X....X.............X............X......X.....X..X.] duration:
46.695111103s trace:
https://jaeger.dev.storj.io/trace/6b00f0e5b917eb2f
FAIL (2026-05-16 19:43:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 46.695122995s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
8.246159417s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:43:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 8.246183904s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [.X..XX....X...XX..X.......XX.X..X.X....X.........X..........] duration:
3.889442633s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:43:13 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.889465275s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
4.182377926s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:42:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.182394066s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [......X....X....X.............X............X......X.....X..X] duration:
1m0.000883755s trace:
https://jaeger.dev.storj.io/trace/6b00f0e5b917eb2f
FAIL (2026-05-16 19:42:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000908653s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
7.683824131s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:42:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 7.683849258s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [..X..XX....X...XX..X.......XX.X..X.X....X.........X.........] duration:
4.581538183s trace:
https://jaeger.dev.storj.io/trace/12e1a08f0c6fd0f4
FAIL (2026-05-16 19:42:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.581569904s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
4.626970346s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:41:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.62699414s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
7.113329011s trace:
https://jaeger.dev.storj.io/trace/6014628d7fc47a45
FAIL (2026-05-16 19:41:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 16 19:35:24.425481025 (384h47m34.94630477s) [upload=368743]: Starting upload
May 16 19:35:24.425549866 (384h47m34.946373611s) [upload=368743]: Done starting upload
May 16 19:35:24.425626924 (384h47m34.946450669s) [upload=368743]: Uploading segments...
May 16 19:35:24.425767021 (384h47m34.946590766s) [upload=368743]: Got next segment. Inline: false
May 16 19:35:24.425876990 (384h47m34.946700735s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 16 19:35:24.425882431 (384h47m34.946706166s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 16 19:35:24.663258858 (384h47m35.184082603s) [seg_pos={0 0}, upload=368743]: Begin upload segment...
May 16 19:35:24.663303744 (384h47m35.184127489s) [seg_pos={0 0}, upload=368743]: Done begin upload segment.
May 16 19:35:24.663320705 (384h47m35.184144451s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663339923 (384h47m35.184163668s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.663797205 (384h47m35.18462095s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664039938 (384h47m35.184863683s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664329882 (384h47m35.185153627s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664596721 (384h47m35.185420456s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664689468 (384h47m35.185513213s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.664859453 (384h47m35.185683198s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665564397 (384h47m35.186388142s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665604224 (384h47m35.186427969s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665927881 (384h47m35.186751626s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.665978868 (384h47m35.186802613s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666208948 (384h47m35.187032693s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666342141 (384h47m35.187165887s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666494162 (384h47m35.187317907s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:24.666797130 (384h47m35.187620875s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Uploading piece...
May 16 19:35:26.421584374 (384h47m36.942408119s) [upload=368743]: Next returned nil segment
May 16 19:35:26.421596407 (384h47m36.942420152s) [upload=368743]: Waiting for error group managing segments...
May 16 19:35:26.718170466 (384h47m37.238994211s) [noise=true , address=145.79.227.20:4443 , piece_id=H4YYO6D7GV63H3OFQ2FFSUNXTA6LETCOUMKRUCORAMMZJ3Z2ECJA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.733497814 (384h47m37.254321559s) [noise=true , address=145.79.227.22:4511 , piece_id=VW3KDEZXBUKJXE5FUUVDQ5YLBISUDL5KUFFNLBMZ557N2CTFH5EQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736006511 (384h47m37.256830256s) [noise=true , address=145.79.227.6:4263 , piece_id=LWZCGIKLEHNIBMPP4E3BGR2DOTC37FU2FWDCEPNPX3ET75D6LZNQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.736692591 (384h47m37.257516326s) [noise=true , address=145.79.227.43:4158 , piece_id=WRE7N3Y6LKLD6QHQG4EDLFC6O5H7XSOMJQQLAXGKNNRD3UZYG75Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.748855539 (384h47m37.269679274s) [noise=true , address=145.79.227.3:4154 , piece_id=465YSM7L7RCWSXZUTXELX75DUWLL6GU52CCGDOJNRQG7IE5JQ6CQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.751250129 (384h47m37.272073874s) [noise=true , address=77.90.31.14:4661 , piece_id=CDCB4H6IBMGVZTC675PYOMYHUV5D3X5FVFNELOC6LC4V6277D5GQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.763834944 (384h47m37.284658689s) [noise=true , address=145.79.227.4:4191 , piece_id=KSZ22TMZ2HFRAW45AFV2AL7A3LVE45AAUSODAONM4YB7XZSNUEXQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:26.824421747 (384h47m37.345245492s) [noise=true , address=145.79.227.25:4622 , piece_id=437E3SLT74YYW4UHLSANUK5WCDSKJ2J5EDOG2BZIXJCQWUCNZE5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.207240761 (384h47m37.728064506s) [noise=true , address=145.79.227.2:4116 , piece_id=FL5OR6FWSLKRFCELKFXJ3T3VSRIKZEH4MYL7JWAZLPX6CWZABEAQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.268746427 (384h47m37.789570182s) [noise=true , address=45.91.116.8:4048 , piece_id=OMM7BGS6P5H4SS7PJML3NSHF52HUH76YT2YKRKBCIPYL3HDGLCTQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.325110963 (384h47m37.845934708s) [noise=true , address=45.91.116.14:4190 , piece_id=DGPZ2JJXQO6HN7QVHJRU7LGEP2DHQ3PGXF4D254YPG7QI4C42DDA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.493618493 (384h47m38.014442228s) [noise=true , address=145.79.227.17:4975 , piece_id=Z7CYYIO7YNLJUC345IWQJETPL2ST36TJUKNAB27LMIAUD4ET3M5Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958522042 (384h47m38.479345817s) [noise=true , address=145.79.227.52:4446 , piece_id=2UYHFRR2QKMMCJGLG7E54WM6ZBJCR444SFQ7VNKPCNKP2TAHIWSQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: <nil>
May 16 19:35:27.958548162 (384h47m38.479371907s) [seg_pos={0 0}, upload=368743]: Satisfied cohort requirements after 13 pieces
May 16 19:35:27.958779604 (384h47m38.479603349s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958787188 (384h47m38.479610933s) [noise=true , address=77.90.31.18:4807 , piece_id=EPQ5KYFCSLSS6ARIUOYEMZKTERIBHQWGU3GZHUXK2GKZ7LTGIRFQ, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958806516 (384h47m38.479630271s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958856270 (384h47m38.479680015s) [noise=true , address=145.79.227.34:4908 , piece_id=RPLU6KVZW5KGNCWWH56OUTOV5Q2JCU4W7SPWRDTER37L3H3CTDLA, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.958964948 (384h47m38.479788693s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
May 16 19:35:27.958972762 (384h47m38.479796507s) [noise=true , address=45.91.116.9:4265 , piece_id=CG7RUTRXM5KEPJG7BXO2JJAX6R54SANRSVKNMULZU2QA7B6QEZ7Q, seg_pos={0 0}, upload=368743]: Long tail cancellation due to optimal threshold
May 16 19:35:27.959001126 (384h47m38.479824871s) [seg_pos={0 0}, upload=368743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 16 19:35:27.959099394 (384h47m38.479923139s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:27.959105346 (384h47m38.479929101s) [upload=368743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:28.247922192 (384h47m38.768745937s) [upload=368743]: Done uploading segments...
May 16 19:35:28.247941589 (384h47m38.768765334s) [upload=368743]: Upload finished. err: <nil>
Duration: 7.113353979s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
4.766823409s trace:
https://jaeger.dev.storj.io/trace/1fcd4f6b17d1b468
FAIL (2026-05-16 19:41:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
{0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245647373 (384h47m34.766471118s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.245733897 (384h47m34.766557642s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246688209 (384h47m34.767511954s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.246719729 (384h47m34.767543474s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.247561836 (384h47m34.768385581s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.248645645 (384h47m34.76946939s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.249619563 (384h47m34.770443308s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.250601677 (384h47m34.771425422s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.251842745 (384h47m34.7726665s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:24.253211327 (384h47m34.774035082s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Uploading piece...
May 16 19:35:25.063793938 (384h47m35.584617683s) [upload=368742]: Next returned nil segment
May 16 19:35:25.063806963 (384h47m35.584630708s) [upload=368742]: Waiting for error group managing segments...
May 16 19:35:25.450573960 (384h47m35.971397705s) [noise=true , address=185.167.204.206:4063 , piece_id=VKGADUROZVVTV6AN34MHSGG7WBRPRLX45MPLCDSRG2U4Q3JC2OFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.451113249 (384h47m35.971936994s) [noise=true , address=185.167.204.205:4031 , piece_id=DWEB7TAHMRS2POBWSJ66UTJUMKSLQUBWY4XIBA7EEI7KLWLBVNKQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.452392260 (384h47m35.973216005s) [noise=true , address=185.167.204.206:4072 , piece_id=P5RXENUDB6EYY3QEWH2PAJ73WXYHAZ253DE36BKR224EW2ZAUL3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455393387 (384h47m35.976217132s) [noise=true , address=185.167.204.206:4059 , piece_id=WM4GJLH4TPUGO4UBGYGNLV7HAGGZX6OJCGUXBKZRR7KWATQSDVGA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.455604350 (384h47m35.976428095s) [noise=true , address=185.167.204.205:4012 , piece_id=HYZBZ2HRSWSVF5B7IZOPLNPLUH3LQ3DODPARJ4VWCMBA635N6IDA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.456811343 (384h47m35.977635088s) [noise=true , address=185.167.204.206:4064 , piece_id=6OV3ZDDP2Y2UDWSLFY43JSZPFQ577CMYIDZ3WSWUBLIW6EXKRZJA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.458142933 (384h47m35.978966668s) [noise=true , address=185.167.204.205:4030 , piece_id=RTISOGDT6SKHJZYH53ONIL5TDWYD7KWTC3JOY4DOJHJ7XOSRSIBQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.462930430 (384h47m35.983754165s) [noise=true , address=185.167.204.205:4024 , piece_id=GCOCULJXXB7KEEALR32KJKNGOVY7APCATYDZSJNY5A65N3VJ3DFA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469054004 (384h47m35.989877749s) [noise=true , address=185.167.204.205:4002 , piece_id=IDZPJDOZE4DJZZA7WD2YU3KJFOAXGWA5UMJGCWPTECHW5T72JXLQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.469690969 (384h47m35.990514714s) [noise=true , address=185.167.204.205:4008 , piece_id=WWWV6KKM7PKZBJMN3BJWLNX4EAJKKHPW7J2APZIBDIWAGPSTORBA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.472420848 (384h47m35.993244594s) [noise=true , address=185.167.204.205:4018 , piece_id=KBYYAUACS2OTO7EQBMSTO2HU5XZREZ7UXP4GLFTRZ2I4AGTI2WYA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.473928104 (384h47m35.99475185s) [noise=true , address=185.167.204.205:4007 , piece_id=QSQG4YKJRBPCYRDMNUMC3777BYZ34AWPJHDWAKIBUVH23B2DF7SQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.475261830 (384h47m35.996085575s) [noise=true , address=185.167.204.205:4025 , piece_id=IMVWBZT75TLF5ROL63Q7SBMXMQK5RTQJPKCOAXJ5FD3NMWIDKBOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:25.555269663 (384h47m36.076093408s) [noise=true , address=81.171.14.70:4030 , piece_id=HZQANAVDIWXMEFSNTK3KLXBNP37AQM7O7DKLLSNPUZZW54NPUUOQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.007314655 (384h47m36.5281384s) [noise=true , address=89.149.201.147:4005 , piece_id=DJDOZFCPEJ5CF3PD7C2A2G6B434AEVNRGBZZW5UZFMTQRAUMCQVQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.010142220 (384h47m36.530965965s) [noise=true , address=81.171.14.70:4032 , piece_id=SZDXUUAUGGHIOCJZB5F4HT4EI7M5CIUX6OQISRVUVBDOIUUQORXA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.057324746 (384h47m36.578148491s) [noise=true , address=81.171.14.70:4042 , piece_id=JIRL35BHFFILGDWTTHDBW6VJO6J6SPELMGIQE6KH6G3GKHEMIITQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.126982529 (384h47m36.647806264s) [noise=true , address=89.149.201.147:4010 , piece_id=YZ5TUVKXW7AXMDMNBDG5QRWLRP4UXXT4E7HA7LVYTUV3NWQ2HWUQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.138423530 (384h47m36.659247275s) [noise=true , address=89.149.201.147:4003 , piece_id=TST7SSOVH5VO3FPWDH2QXWYA2DXKJY27LSPQRE356MTGGIX2X2TQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.195283261 (384h47m36.716107006s) [noise=true , address=81.171.14.70:4041 , piece_id=LYJYOENI6KHE36B4RKNR5BXBBEYXAD4I64AGFJLKHZLCN3DPHDNA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.362318842 (384h47m36.883142587s) [noise=true , address=81.171.14.70:4036 , piece_id=R4TV4MMJR2ENYGRU7HSI4MI4U5IAITMAQ7TNUOWH4M6MHSTVEP4Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.377329305 (384h47m36.89815305s) [noise=true , address=81.171.14.70:4027 , piece_id=TTAH7L7SGOVZPTAIGJBPCHMDRZUQSLLOYFCBUM2TV2C4EQ3QOM3Q, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.378116498 (384h47m36.898940243s) [noise=true , address=81.171.14.70:4029 , piece_id=3HC5GQ453RFNFSMVC7ZI2MD3PLQSCY4U4TGIDFTJ47TPPEXWL3QA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.529821725 (384h47m37.05064547s) [noise=true , address=89.149.201.147:4006 , piece_id=DROJWGJXZ76XA3ZB6YYZM4OVYZZFZEHSWHJHKGBFANJP6BHXLSJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.560988347 (384h47m37.081812092s) [noise=true , address=89.149.201.147:4020 , piece_id=BAFOIFHCPRLB5ZVUDNXK5ETP5QAUZS5UKGCSEUWXLHWY34CZIEJQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670064319 (384h47m37.190888064s) [noise=true , address=89.149.201.147:4014 , piece_id=PP24GLUODGXQBCCGRZMIDOPFQJSGSOELSZZKPIDO3QBTLKIL7TRA, seg_pos={0 0}, upload=368742]: Done uploading piece. err: <nil>
May 16 19:35:26.670082995 (384h47m37.19090673s) [seg_pos={0 0}, upload=368742]: Satisfied cohort requirements after 26 pieces
May 16 19:35:26.670238612 (384h47m37.191062347s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHiz99nonr7kwp4RLRvZdvKUJW3gVB3tx5vpDDbfijn2qrF2m): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670247017 (384h47m37.191070762s) [noise=true , address=81.171.14.70:4047 , piece_id=LNDFYFFGJLOPJXQX5RAZBLP3BUHMALZMJB3T2OK4M4EQ2EXU5OMQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670402003 (384h47m37.191225738s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF
May 16 19:35:26.670424717 (384h47m37.191248462s) [noise=true , address=89.149.201.147:4023 , piece_id=BZZ4Q3TDTWJZNBENHSSY6IF47ONHBZZ2BRDLKRUZXOUEOXH5ILIQ, seg_pos={0 0}, upload=368742]: Long tail cancellation due to optimal threshold
May 16 19:35:26.670454473 (384h47m37.191278218s) [seg_pos={0 0}, upload=368742]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
May 16 19:35:26.670555646 (384h47m37.191379381s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 16 19:35:26.670561237 (384h47m37.191384983s) [upload=368742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 16 19:35:26.906606223 (384h47m37.427429968s) [upload=368742]: Done uploading segments...
May 16 19:35:26.906651069 (384h47m37.427474804s) [upload=368742]: Upload finished. err: <nil>
Duration: 4.766866491s
core
uplink
eu-select-1
eu1-placement-15
|