Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X.....] duration:
5.000473361s trace:
https://jaeger.dev.storj.io/trace/2285969319795cce
FAIL (2025-12-26 15:16:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000500093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X..............] duration:
54.26031541s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:16:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 54.260342862s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X....X.XX.XXX...XX...........XX..X.X.....X.....X.X..........] duration:
5.779371799s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:16:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.779399232s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X....] duration:
3.709084776s trace:
https://jaeger.dev.storj.io/trace/2285969319795cce
FAIL (2025-12-26 15:16:08 UTC)
orbiter warning: context deadline exceeded
Duration: 3.709116036s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X...] duration:
4.280630128s trace:
https://jaeger.dev.storj.io/trace/2285969319795cce
FAIL (2025-12-26 15:15:31 UTC)
orbiter warning: context deadline exceeded
Duration: 4.280669263s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X.........] duration:
5.815553694s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:15:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.81558802s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
22 out of 60 is failed (threshold=3) [XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X.............] duration:
45.317785884s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:15:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 45.317875584s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
22 out of 60 is failed (threshold=3) [.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X............] duration:
49.354943118s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:14:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 49.354963476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X..] duration:
5.00124916s trace:
https://jaeger.dev.storj.io/trace/2285969319795cce
FAIL (2025-12-26 15:14:30 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001269338s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X........] duration:
5.958831759s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:14:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.958862877s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X.] duration:
5.001252392s trace:
https://jaeger.dev.storj.io/trace/2285969319795cce
FAIL (2025-12-26 15:14:05 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001285215s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [X.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X...........] duration:
48.797641706s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:13:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 48.797657135s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X.......] duration:
6.283196279s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:13:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 6.283234822s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X..........] duration:
53.324230937s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:13:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 53.324249511s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X] duration:
5.000883165s trace:
https://jaeger.dev.storj.io/trace/2285969319795cce
FAIL (2025-12-26 15:12:59 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000909765s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.] duration:
4.639560795s trace:
https://jaeger.dev.storj.io/trace/c3a1b0c39b6a564
FAIL (2025-12-26 15:12:40 UTC)
orbiter warning: context deadline exceeded
Duration: 4.639603586s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X......] duration:
5.910587902s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:12:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.910616507s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X] duration:
5.000854369s trace:
https://jaeger.dev.storj.io/trace/c3a1b0c39b6a564
FAIL (2025-12-26 15:12:11 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000879007s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X.........] duration:
53.836669485s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:12:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 53.836689443s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X.....] duration:
5.861804379s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:11:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.861838746s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
15.255419956s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:11:51 UTC)
orbiter warning: context deadline exceeded
Duration: 15.255450032s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.] duration:
4.943938864s trace:
https://jaeger.dev.storj.io/trace/423252f474d0b167
FAIL (2025-12-26 15:11:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.943962709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
17.229097145s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:11:20 UTC)
orbiter warning: context deadline exceeded
Duration: 17.229186716s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X........] duration:
58.026462695s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:11:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 58.026484405s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X....] duration:
5.806917836s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:11:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.806943344s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X] duration:
5.001383187s trace:
https://jaeger.dev.storj.io/trace/423252f474d0b167
FAIL (2025-12-26 15:10:46 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001406613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X..XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X...] duration:
5.843507177s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:10:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.843535712s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
11.774303949s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:10:16 UTC)
orbiter warning: context deadline exceeded
Duration: 11.774384403s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [..XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X.......] duration:
50.491955934s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:10:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 50.491973306s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
15.665027096s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:09:47 UTC)
orbiter warning: context deadline exceeded
Duration: 15.66506036s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [XX..XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X..] duration:
5.990384967s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:09:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.990412982s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...] duration:
3.553340704s trace:
https://jaeger.dev.storj.io/trace/1a3d63e1175cdf5f
FAIL (2025-12-26 15:09:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.553369169s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [X.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X..] duration:
3.491280872s trace:
https://jaeger.dev.storj.io/trace/1a3d63e1175cdf5f
FAIL (2025-12-26 15:09:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.491311189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [...XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X......] duration:
55.123531987s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:09:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 55.123563187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X.] duration:
5.000498031s trace:
https://jaeger.dev.storj.io/trace/1a3d63e1175cdf5f
FAIL (2025-12-26 15:08:55 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000523649s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [XXX..XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X.] duration:
5.880190899s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:08:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 5.880241214s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
8.874382948s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:08:50 UTC)
orbiter warning: context deadline exceeded
Duration: 8.87440554s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.X] duration:
30.005362311s trace:
https://jaeger.dev.storj.io/trace/234e4fdb6ee10552
FAIL (2025-12-26 15:08:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
oad=441976]: Uploading segments...
Dec 26 15:08:08.634126522 (503h39m8.505209567s) [upload=441976]: Got next segment. Inline: false
Dec 26 15:08:08.634168923 (503h39m8.505251958s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:08:08.634173411 (503h39m8.505256446s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:08:08.921709449 (503h39m8.792792494s) [seg_pos={0 0}, upload=441976]: Begin upload segment...
Dec 26 15:08:08.921765777 (503h39m8.792848812s) [seg_pos={0 0}, upload=441976]: Done begin upload segment.
Dec 26 15:08:08.921831282 (503h39m8.792914327s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.921885795 (503h39m8.79296884s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.922758326 (503h39m8.793841372s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923236467 (503h39m8.794319502s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.923676375 (503h39m8.79475942s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.924121453 (503h39m8.795204498s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.926415881 (503h39m8.797498916s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.928170793 (503h39m8.799253838s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.932021646 (503h39m8.803104681s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.936261489 (503h39m8.807344534s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.939541004 (503h39m8.81062405s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.941777162 (503h39m8.812860207s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.944200867 (503h39m8.815283902s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.945752572 (503h39m8.816835607s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.949475591 (503h39m8.820558636s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:08.950837304 (503h39m8.821920369s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Uploading piece...
Dec 26 15:08:10.552891913 (503h39m10.423974948s) [upload=441976]: Next returned nil segment
Dec 26 15:08:10.552909787 (503h39m10.423992832s) [upload=441976]: Waiting for error group managing segments...
Dec 26 15:08:10.941353821 (503h39m10.812436866s) [noise=true , address=145.79.227.50:4374 , piece_id=2PNGBFMTCMNT5MAZOPQL6PJ3ZP6FJJMCSVPPZVQ4TODFNODR4DUA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.942444217 (503h39m10.813527252s) [noise=true , address=145.79.227.13:4803 , piece_id=AJC223Y2WDHBRRT3E3JXCV4Q3KLZEBLTOMJWAPHPLN4WOZJM4IRA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.953679792 (503h39m10.824762837s) [noise=true , address=145.79.227.45:4230 , piece_id=NPTJ2MZYDAYJTEBZ5EK3Z2PS4MNHXB3YTEJBFG7QAJCSYZ5M7AMQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954227295 (503h39m10.82531034s) [noise=true , address=145.79.227.41:4087 , piece_id=AASP6N6W4Y7ZV3JWWOOXHAYEFF2TGFA3E6DFWFCNRW6BW7KGF7LA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.954335761 (503h39m10.825418796s) [noise=true , address=145.79.227.46:4601 , piece_id=BOQMJPMIYMOI32J44GWGGSRAZ5Q2ZSOKDJVZLDCQE5UUSYVJ63PQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.955044250 (503h39m10.826127285s) [noise=true , address=145.79.227.47:4266 , piece_id=BHAC3LW6YWJKTUZB727GZOSBTPK64SKEKWSURW5OUSLRMKV4Z4RQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.973346881 (503h39m10.844429916s) [noise=true , address=145.79.227.15:4876 , piece_id=3KOIRQ3CN4LC3QRQE2RTDOOUW4I4KONSVOJRFKQXA3QW2DGR65PA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.986613122 (503h39m10.857696157s) [noise=true , address=145.79.227.4:4185 , piece_id=MMF77KRXQ6JABALLUK4KBLL5A6KY5IAYM7HA276AMXUI6F4NEEJQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:10.988649240 (503h39m10.859732275s) [noise=true , address=145.79.227.52:4447 , piece_id=NVFJDO7NIAJIIVHEMZ2UVPC64USXJAE2GOZN7IV265GT52NTB4BA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.184491628 (503h39m11.055574673s) [noise=true , address=45.91.116.4:4940 , piece_id=7AKCJOPQW7K4LME3C2VZ26SKU35Q4Y4V4BSTWGV5TCF5G63LQONA, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.216326617 (503h39m11.087409652s) [noise=true , address=45.91.116.11:4118 , piece_id=J76YOQTGKYNXQJ5PJX3GH4CW22XCDEQKDW4SXCDYPF4AOIY2LXSQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.278643509 (503h39m11.149726554s) [noise=true , address=45.91.116.13:4156 , piece_id=FWE2J2N57V5FJYNSHTFUDHQZ3LRVSKR3PYIOLYFJBPGOIV7TGUQQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618807770 (503h39m11.489890805s) [noise=true , address=45.91.116.12:4225 , piece_id=LVST67AQ4OIDWMLCFHJ3TQDUIAOMEMGMXLPRMPN4LRUAOIN7J26Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: <nil>
Dec 26 15:08:11.618832578 (503h39m11.489915613s) [seg_pos={0 0}, upload=441976]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:08:11.619086321 (503h39m11.490169366s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50872->45.91.116.6:4973: use of closed network connection
Dec 26 15:08:11.619094026 (503h39m11.490177071s) [noise=true , address=45.91.116.6:4973 , piece_id=4ND3CTWZ63WJJ5KUBOTB7GCSB5SQVRQQL6RLTFASFCVML5APGNNQ, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619174239 (503h39m11.490257294s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 15:08:11.619196521 (503h39m11.490279566s) [noise=true , address=77.90.31.12:4590 , piece_id=BK4MWVXLBVMLNXMSAKNI7YAZWG5CB6ONPKGXK3OJH5KDUUINDP5Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619239503 (503h39m11.490322538s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51646->45.91.116.7:4010: use of closed network connection
Dec 26 15:08:11.619246456 (503h39m11.490329501s) [noise=true , address=45.91.116.7:4010 , piece_id=3XCOFKFVNKHUT2Y3IQTNXZSYLVH6SH6YTJ5KEJGZVD6QN2NJIO7Q, seg_pos={0 0}, upload=441976]: Long tail cancellation due to optimal threshold
Dec 26 15:08:11.619338191 (503h39m11.490421226s) [seg_pos={0 0}, upload=441976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:08:11.619420678 (503h39m11.490503723s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:08:11.619425397 (503h39m11.490508442s) [upload=441976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:08:11.910424089 (503h39m11.781507134s) [upload=441976]: Done uploading segments...
Dec 26 15:08:11.910447894 (503h39m11.781530939s) [upload=441976]: Upload finished. err: <nil>
Duration: 30.005386648s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X] duration:
5.001643373s trace:
https://jaeger.dev.storj.io/trace/1a3d63e1175cdf5f
FAIL (2025-12-26 15:08:29 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001663721s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X.....] duration:
49.625550329s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:08:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 49.625568654s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
15.278060067s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:07:51 UTC)
orbiter warning: context deadline exceeded
Duration: 15.278089484s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..] duration:
5.001009036s trace:
https://jaeger.dev.storj.io/trace/436d7985d73d7cb8
FAIL (2025-12-26 15:07:46 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001023743s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X.] duration:
6.239591641s trace:
https://jaeger.dev.storj.io/trace/1b64f037589c1d0f
FAIL (2025-12-26 15:07:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UPTYG3KDRWUOCO6WEA2DNP2PELB2RFLFALFYCZECYPJFCJ5MY6EA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UPQ6XCK2CXONVIK5IF7EIDZO76XVJ4PERAPWJWMDLSQ36MM7KAXA) context deadline exceeded
======LOGS======
(503h37m50.739733965s) [upload=441961]: Done starting upload
Dec 26 15:06:50.868861311 (503h37m50.739944356s) [upload=441961]: Uploading segments...
Dec 26 15:06:50.868926365 (503h37m50.74000941s) [upload=441961]: Got next segment. Inline: false
Dec 26 15:06:50.868958186 (503h37m50.740041231s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:06:50.868962864 (503h37m50.740045909s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:06:51.119492327 (503h37m50.990575372s) [seg_pos={0 0}, upload=441961]: Begin upload segment...
Dec 26 15:06:51.119541981 (503h37m50.990625026s) [seg_pos={0 0}, upload=441961]: Done begin upload segment.
Dec 26 15:06:51.119584381 (503h37m50.990667416s) [noise=true , address=145.79.227.41:4086 , piece_id=LSSJQSRD4WWIHGEQB356HLVE2RAXUGSWFYU3Y6FFBERN5WW4KB6A, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.119627023 (503h37m50.990710068s) [noise=true , address=77.90.31.21:4880 , piece_id=UPQ6XCK2CXONVIK5IF7EIDZO76XVJ4PERAPWJWMDLSQ36MM7KAXA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.120119651 (503h37m50.991202696s) [noise=true , address=45.91.116.11:4118 , piece_id=VU5RFWTQQLSBYKY7SRUFC7TWJJOFNZW45MLT32XF2YDTY65TV4KQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.120535002 (503h37m50.991618037s) [noise=true , address=145.79.227.50:4374 , piece_id=V25O7US54M547XHVKF7W75E7W2CT5IRKWJHJPV7H4BUSEYEKSXCQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.120904064 (503h37m50.991987099s) [noise=true , address=145.79.227.51:4411 , piece_id=O4KKK4LQXUNYR5ZW6WSHVNEZJS7MPAJA3UTQ75F43GYYWRPAFXVA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.121324024 (503h37m50.992407069s) [noise=true , address=45.91.116.8:4046 , piece_id=LVP74FUXL7BVOBHFKGTFI6V674H27GAMOBT5B3NTQ7LC2RKUMU7Q, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.121629015 (503h37m50.99271205s) [noise=true , address=145.79.227.46:4601 , piece_id=GOGF6BCUQSR2ZBVZSQDTFZMLFC4JQV24W7DS3ZP5UZG7HPGJI7FQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.122086296 (503h37m50.993169341s) [noise=true , address=45.91.116.5:4300 , piece_id=ROEV66DFYG4UBBU5BCTMBFK6GLOGSKJZIANIO7N2B5EKCV5KULVA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.122616655 (503h37m50.9936997s) [noise=true , address=45.91.116.7:4010 , piece_id=UPTYG3KDRWUOCO6WEA2DNP2PELB2RFLFALFYCZECYPJFCJ5MY6EA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.123771215 (503h37m50.99485426s) [noise=true , address=145.79.227.37:4566 , piece_id=333FDNCIS6YTIMIWZ5HKCNR37HMRUOKQ3MLFFDDXAB56MDQTSNIA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.126457329 (503h37m50.997540374s) [noise=true , address=145.79.227.14:4840 , piece_id=NAQICV2ZNVU3UMUGGB5WJIEAIJ4OT4VSHTBJZZDVPTCS6G36OJNQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.132183052 (503h37m51.003266097s) [noise=true , address=145.79.227.40:4050 , piece_id=TRPEJXG27LFHORKK4UM5ABQJOV3IMB4PP4WLTPBZALUTE7GKQVLA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.134970119 (503h37m51.006053164s) [noise=true , address=145.79.227.49:4339 , piece_id=CU227AT5WWAF3V2SPMCGLDZVT2NDTP7CHQX7JQEP46HKGAHJYIRQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.136727846 (503h37m51.007810881s) [noise=true , address=145.79.227.8:4330 , piece_id=MP57YGPGCVA6RP7XGJ3WXIVRNZ62CSRJ2PUKTOBMWOAJQ67TIHVA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.137366201 (503h37m51.008449246s) [noise=true , address=145.79.227.19:4078 , piece_id=FV6CPVRT6QZRNSNFVHVMBECE7IS45X7NNWD242P6AF3EMS2SS4OA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.142879180 (503h37m51.013962225s) [noise=true , address=145.79.227.52:4445 , piece_id=UB7ZSVK436YFZPQ7JXG6XSLDQNIIV5RILBK7BLISVRDNPIXBEWQA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:52.843386871 (503h37m52.714469916s) [upload=441961]: Next returned nil segment
Dec 26 15:06:52.843400406 (503h37m52.714483451s) [upload=441961]: Waiting for error group managing segments...
Dec 26 15:06:53.129454964 (503h37m53.000538009s) [noise=true , address=145.79.227.50:4374 , piece_id=V25O7US54M547XHVKF7W75E7W2CT5IRKWJHJPV7H4BUSEYEKSXCQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.138829224 (503h37m53.009912269s) [noise=true , address=145.79.227.37:4566 , piece_id=333FDNCIS6YTIMIWZ5HKCNR37HMRUOKQ3MLFFDDXAB56MDQTSNIA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.139359774 (503h37m53.010442819s) [noise=true , address=145.79.227.46:4601 , piece_id=GOGF6BCUQSR2ZBVZSQDTFZMLFC4JQV24W7DS3ZP5UZG7HPGJI7FQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.139660136 (503h37m53.010743171s) [noise=true , address=145.79.227.14:4840 , piece_id=NAQICV2ZNVU3UMUGGB5WJIEAIJ4OT4VSHTBJZZDVPTCS6G36OJNQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.145855994 (503h37m53.016939029s) [noise=true , address=145.79.227.51:4411 , piece_id=O4KKK4LQXUNYR5ZW6WSHVNEZJS7MPAJA3UTQ75F43GYYWRPAFXVA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.147825584 (503h37m53.018908629s) [noise=true , address=145.79.227.52:4445 , piece_id=UB7ZSVK436YFZPQ7JXG6XSLDQNIIV5RILBK7BLISVRDNPIXBEWQA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.150031866 (503h37m53.021114911s) [noise=true , address=145.79.227.19:4078 , piece_id=FV6CPVRT6QZRNSNFVHVMBECE7IS45X7NNWD242P6AF3EMS2SS4OA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.150681663 (503h37m53.021764708s) [noise=true , address=145.79.227.8:4330 , piece_id=MP57YGPGCVA6RP7XGJ3WXIVRNZ62CSRJ2PUKTOBMWOAJQ67TIHVA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.159439309 (503h37m53.030522354s) [noise=true , address=145.79.227.41:4086 , piece_id=LSSJQSRD4WWIHGEQB356HLVE2RAXUGSWFYU3Y6FFBERN5WW4KB6A, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.168083549 (503h37m53.039166594s) [noise=true , address=145.79.227.49:4339 , piece_id=CU227AT5WWAF3V2SPMCGLDZVT2NDTP7CHQX7JQEP46HKGAHJYIRQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.303823662 (503h37m53.174906697s) [noise=true , address=77.90.31.21:4880 , piece_id=UPQ6XCK2CXONVIK5IF7EIDZO76XVJ4PERAPWJWMDLSQ36MM7KAXA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.423120664 (503h37m53.294203699s) [noise=true , address=45.91.116.7:4010 , piece_id=UPTYG3KDRWUOCO6WEA2DNP2PELB2RFLFALFYCZECYPJFCJ5MY6EA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.579634744 (503h37m53.450717809s) [noise=true , address=45.91.116.8:4046 , piece_id=LVP74FUXL7BVOBHFKGTFI6V674H27GAMOBT5B3NTQ7LC2RKUMU7Q, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.579668779 (503h37m53.450751834s) [seg_pos={0 0}, upload=441961]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:06:53.579940766 (503h37m53.451023811s) [noise=true , address=145.79.227.40:4050 , piece_id=TRPEJXG27LFHORKK4UM5ABQJOV3IMB4PP4WLTPBZALUTE7GKQVLA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 26 15:06:53.579953370 (503h37m53.451036415s) [noise=true , address=145.79.227.40:4050 , piece_id=TRPEJXG27LFHORKK4UM5ABQJOV3IMB4PP4WLTPBZALUTE7GKQVLA, seg_pos={0 0}, upload=441961]: Long tail cancellation due to optimal threshold
Dec 26 15:06:53.580014297 (503h37m53.451097362s) [noise=true , address=45.91.116.5:4300 , piece_id=ROEV66DFYG4UBBU5BCTMBFK6GLOGSKJZIANIO7N2B5EKCV5KULVA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:41596->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:41596->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41596->45.91.116.5:4300: use of closed network connection
Dec 26 15:06:53.580036007 (503h37m53.451119052s) [noise=true , address=45.91.116.5:4300 , piece_id=ROEV66DFYG4UBBU5BCTMBFK6GLOGSKJZIANIO7N2B5EKCV5KULVA, seg_pos={0 0}, upload=441961]: Long tail cancellation due to optimal threshold
Dec 26 15:06:53.580188618 (503h37m53.451271664s) [noise=true , address=45.91.116.11:4118 , piece_id=VU5RFWTQQLSBYKY7SRUFC7TWJJOFNZW45MLT32XF2YDTY65TV4KQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:59140->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 15:06:53.580197385 (503h37m53.45128043s) [noise=true , address=45.91.116.11:4118 , piece_id=VU5RFWTQQLSBYKY7SRUFC7TWJJOFNZW45MLT32XF2YDTY65TV4KQ, seg_pos={0 0}, upload=441961]: Long tail cancellation due to optimal threshold
Dec 26 15:06:53.580225439 (503h37m53.451308474s) [seg_pos={0 0}, upload=441961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:06:53.580305681 (503h37m53.451388726s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:06:53.580311682 (503h37m53.451394727s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:06:53.869696612 (503h37m53.740779657s) [upload=441961]: Done uploading segments...
Dec 26 15:06:53.869719896 (503h37m53.740802941s) [upload=441961]: Upload finished. err: <nil>
Duration: 6.239614794s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X....] duration:
56.540461516s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:07:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 56.540477116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X.....X.....X] duration:
30.002418399s trace:
https://jaeger.dev.storj.io/trace/1b64f037589c1d0f
FAIL (2025-12-26 15:07:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UPTYG3KDRWUOCO6WEA2DNP2PELB2RFLFALFYCZECYPJFCJ5MY6EA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UPQ6XCK2CXONVIK5IF7EIDZO76XVJ4PERAPWJWMDLSQ36MM7KAXA) context deadline exceeded
======LOGS======
(503h37m50.739733965s) [upload=441961]: Done starting upload
Dec 26 15:06:50.868861311 (503h37m50.739944356s) [upload=441961]: Uploading segments...
Dec 26 15:06:50.868926365 (503h37m50.74000941s) [upload=441961]: Got next segment. Inline: false
Dec 26 15:06:50.868958186 (503h37m50.740041231s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 15:06:50.868962864 (503h37m50.740045909s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 15:06:51.119492327 (503h37m50.990575372s) [seg_pos={0 0}, upload=441961]: Begin upload segment...
Dec 26 15:06:51.119541981 (503h37m50.990625026s) [seg_pos={0 0}, upload=441961]: Done begin upload segment.
Dec 26 15:06:51.119584381 (503h37m50.990667416s) [noise=true , address=145.79.227.41:4086 , piece_id=LSSJQSRD4WWIHGEQB356HLVE2RAXUGSWFYU3Y6FFBERN5WW4KB6A, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.119627023 (503h37m50.990710068s) [noise=true , address=77.90.31.21:4880 , piece_id=UPQ6XCK2CXONVIK5IF7EIDZO76XVJ4PERAPWJWMDLSQ36MM7KAXA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.120119651 (503h37m50.991202696s) [noise=true , address=45.91.116.11:4118 , piece_id=VU5RFWTQQLSBYKY7SRUFC7TWJJOFNZW45MLT32XF2YDTY65TV4KQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.120535002 (503h37m50.991618037s) [noise=true , address=145.79.227.50:4374 , piece_id=V25O7US54M547XHVKF7W75E7W2CT5IRKWJHJPV7H4BUSEYEKSXCQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.120904064 (503h37m50.991987099s) [noise=true , address=145.79.227.51:4411 , piece_id=O4KKK4LQXUNYR5ZW6WSHVNEZJS7MPAJA3UTQ75F43GYYWRPAFXVA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.121324024 (503h37m50.992407069s) [noise=true , address=45.91.116.8:4046 , piece_id=LVP74FUXL7BVOBHFKGTFI6V674H27GAMOBT5B3NTQ7LC2RKUMU7Q, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.121629015 (503h37m50.99271205s) [noise=true , address=145.79.227.46:4601 , piece_id=GOGF6BCUQSR2ZBVZSQDTFZMLFC4JQV24W7DS3ZP5UZG7HPGJI7FQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.122086296 (503h37m50.993169341s) [noise=true , address=45.91.116.5:4300 , piece_id=ROEV66DFYG4UBBU5BCTMBFK6GLOGSKJZIANIO7N2B5EKCV5KULVA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.122616655 (503h37m50.9936997s) [noise=true , address=45.91.116.7:4010 , piece_id=UPTYG3KDRWUOCO6WEA2DNP2PELB2RFLFALFYCZECYPJFCJ5MY6EA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.123771215 (503h37m50.99485426s) [noise=true , address=145.79.227.37:4566 , piece_id=333FDNCIS6YTIMIWZ5HKCNR37HMRUOKQ3MLFFDDXAB56MDQTSNIA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.126457329 (503h37m50.997540374s) [noise=true , address=145.79.227.14:4840 , piece_id=NAQICV2ZNVU3UMUGGB5WJIEAIJ4OT4VSHTBJZZDVPTCS6G36OJNQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.132183052 (503h37m51.003266097s) [noise=true , address=145.79.227.40:4050 , piece_id=TRPEJXG27LFHORKK4UM5ABQJOV3IMB4PP4WLTPBZALUTE7GKQVLA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.134970119 (503h37m51.006053164s) [noise=true , address=145.79.227.49:4339 , piece_id=CU227AT5WWAF3V2SPMCGLDZVT2NDTP7CHQX7JQEP46HKGAHJYIRQ, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.136727846 (503h37m51.007810881s) [noise=true , address=145.79.227.8:4330 , piece_id=MP57YGPGCVA6RP7XGJ3WXIVRNZ62CSRJ2PUKTOBMWOAJQ67TIHVA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.137366201 (503h37m51.008449246s) [noise=true , address=145.79.227.19:4078 , piece_id=FV6CPVRT6QZRNSNFVHVMBECE7IS45X7NNWD242P6AF3EMS2SS4OA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:51.142879180 (503h37m51.013962225s) [noise=true , address=145.79.227.52:4445 , piece_id=UB7ZSVK436YFZPQ7JXG6XSLDQNIIV5RILBK7BLISVRDNPIXBEWQA, seg_pos={0 0}, upload=441961]: Uploading piece...
Dec 26 15:06:52.843386871 (503h37m52.714469916s) [upload=441961]: Next returned nil segment
Dec 26 15:06:52.843400406 (503h37m52.714483451s) [upload=441961]: Waiting for error group managing segments...
Dec 26 15:06:53.129454964 (503h37m53.000538009s) [noise=true , address=145.79.227.50:4374 , piece_id=V25O7US54M547XHVKF7W75E7W2CT5IRKWJHJPV7H4BUSEYEKSXCQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.138829224 (503h37m53.009912269s) [noise=true , address=145.79.227.37:4566 , piece_id=333FDNCIS6YTIMIWZ5HKCNR37HMRUOKQ3MLFFDDXAB56MDQTSNIA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.139359774 (503h37m53.010442819s) [noise=true , address=145.79.227.46:4601 , piece_id=GOGF6BCUQSR2ZBVZSQDTFZMLFC4JQV24W7DS3ZP5UZG7HPGJI7FQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.139660136 (503h37m53.010743171s) [noise=true , address=145.79.227.14:4840 , piece_id=NAQICV2ZNVU3UMUGGB5WJIEAIJ4OT4VSHTBJZZDVPTCS6G36OJNQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.145855994 (503h37m53.016939029s) [noise=true , address=145.79.227.51:4411 , piece_id=O4KKK4LQXUNYR5ZW6WSHVNEZJS7MPAJA3UTQ75F43GYYWRPAFXVA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.147825584 (503h37m53.018908629s) [noise=true , address=145.79.227.52:4445 , piece_id=UB7ZSVK436YFZPQ7JXG6XSLDQNIIV5RILBK7BLISVRDNPIXBEWQA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.150031866 (503h37m53.021114911s) [noise=true , address=145.79.227.19:4078 , piece_id=FV6CPVRT6QZRNSNFVHVMBECE7IS45X7NNWD242P6AF3EMS2SS4OA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.150681663 (503h37m53.021764708s) [noise=true , address=145.79.227.8:4330 , piece_id=MP57YGPGCVA6RP7XGJ3WXIVRNZ62CSRJ2PUKTOBMWOAJQ67TIHVA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.159439309 (503h37m53.030522354s) [noise=true , address=145.79.227.41:4086 , piece_id=LSSJQSRD4WWIHGEQB356HLVE2RAXUGSWFYU3Y6FFBERN5WW4KB6A, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.168083549 (503h37m53.039166594s) [noise=true , address=145.79.227.49:4339 , piece_id=CU227AT5WWAF3V2SPMCGLDZVT2NDTP7CHQX7JQEP46HKGAHJYIRQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.303823662 (503h37m53.174906697s) [noise=true , address=77.90.31.21:4880 , piece_id=UPQ6XCK2CXONVIK5IF7EIDZO76XVJ4PERAPWJWMDLSQ36MM7KAXA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.423120664 (503h37m53.294203699s) [noise=true , address=45.91.116.7:4010 , piece_id=UPTYG3KDRWUOCO6WEA2DNP2PELB2RFLFALFYCZECYPJFCJ5MY6EA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.579634744 (503h37m53.450717809s) [noise=true , address=45.91.116.8:4046 , piece_id=LVP74FUXL7BVOBHFKGTFI6V674H27GAMOBT5B3NTQ7LC2RKUMU7Q, seg_pos={0 0}, upload=441961]: Done uploading piece. err: <nil>
Dec 26 15:06:53.579668779 (503h37m53.450751834s) [seg_pos={0 0}, upload=441961]: Segment reached optimal threshold of 13 pieces.
Dec 26 15:06:53.579940766 (503h37m53.451023811s) [noise=true , address=145.79.227.40:4050 , piece_id=TRPEJXG27LFHORKK4UM5ABQJOV3IMB4PP4WLTPBZALUTE7GKQVLA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 26 15:06:53.579953370 (503h37m53.451036415s) [noise=true , address=145.79.227.40:4050 , piece_id=TRPEJXG27LFHORKK4UM5ABQJOV3IMB4PP4WLTPBZALUTE7GKQVLA, seg_pos={0 0}, upload=441961]: Long tail cancellation due to optimal threshold
Dec 26 15:06:53.580014297 (503h37m53.451097362s) [noise=true , address=45.91.116.5:4300 , piece_id=ROEV66DFYG4UBBU5BCTMBFK6GLOGSKJZIANIO7N2B5EKCV5KULVA, seg_pos={0 0}, upload=441961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:41596->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:41596->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41596->45.91.116.5:4300: use of closed network connection
Dec 26 15:06:53.580036007 (503h37m53.451119052s) [noise=true , address=45.91.116.5:4300 , piece_id=ROEV66DFYG4UBBU5BCTMBFK6GLOGSKJZIANIO7N2B5EKCV5KULVA, seg_pos={0 0}, upload=441961]: Long tail cancellation due to optimal threshold
Dec 26 15:06:53.580188618 (503h37m53.451271664s) [noise=true , address=45.91.116.11:4118 , piece_id=VU5RFWTQQLSBYKY7SRUFC7TWJJOFNZW45MLT32XF2YDTY65TV4KQ, seg_pos={0 0}, upload=441961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:59140->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 15:06:53.580197385 (503h37m53.45128043s) [noise=true , address=45.91.116.11:4118 , piece_id=VU5RFWTQQLSBYKY7SRUFC7TWJJOFNZW45MLT32XF2YDTY65TV4KQ, seg_pos={0 0}, upload=441961]: Long tail cancellation due to optimal threshold
Dec 26 15:06:53.580225439 (503h37m53.451308474s) [seg_pos={0 0}, upload=441961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 15:06:53.580305681 (503h37m53.451388726s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:06:53.580311682 (503h37m53.451394727s) [upload=441961]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:06:53.869696612 (503h37m53.740779657s) [upload=441961]: Done uploading segments...
Dec 26 15:06:53.869719896 (503h37m53.740802941s) [upload=441961]: Upload finished. err: <nil>
Duration: 30.002446823s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
9.949988767s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:07:05 UTC)
orbiter warning: context deadline exceeded
Duration: 9.950015208s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
16.757951322s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:06:41 UTC)
orbiter warning: context deadline exceeded
Duration: 16.757976432s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX.] duration:
5.002048725s trace:
https://jaeger.dev.storj.io/trace/436d7985d73d7cb8
FAIL (2025-12-26 15:06:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00207755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX.X...] duration:
54.79417784s trace:
https://jaeger.dev.storj.io/trace/7250569a9bf58122
FAIL (2025-12-26 15:06:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828173942 (503h33m23.699256987s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828559106 (503h33m23.699642151s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828628969 (503h33m23.699712004s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.828989525 (503h33m23.70007257s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.829502903 (503h33m23.700585938s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830071685 (503h33m23.70115473s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830488399 (503h33m23.701571444s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.830882670 (503h33m23.701965705s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831145821 (503h33m23.702228856s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831568526 (503h33m23.702651571s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831655231 (503h33m23.702738276s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.831794566 (503h33m23.702877602s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832350545 (503h33m23.70343359s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.832457178 (503h33m23.703540213s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:23.833230871 (503h33m23.704313906s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Uploading piece...
Dec 26 15:02:25.197673898 (503h33m25.068756943s) [upload=441894]: Next returned nil segment
Dec 26 15:02:25.197693055 (503h33m25.06877609s) [upload=441894]: Waiting for error group managing segments...
Dec 26 15:02:25.519802467 (503h33m25.390885522s) [noise=true , address=64.49.13.140:4507 , piece_id=HS3VBRN5P4BZ35QMWBFIXDGU3Y4HU5PO2PUBYD4TKXLLFWQLVD7A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.526361446 (503h33m25.397444491s) [noise=true , address=64.49.13.27:4968 , piece_id=ZP7WXUEYJPUH5QMA4EMJNIJ7QDOFLU5AWZOGQKEBM3FZWVXOI42A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.565480736 (503h33m25.436563771s) [noise=true , address=64.49.13.139:4472 , piece_id=4V6JD6JOIEY46VVKDOZ7IJHFZPOBAS6TBOYIXXJP6W342OFBNABQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.719029259 (503h33m25.590112304s) [noise=true , address=145.79.227.34:4934 , piece_id=BXKPLQKN3SFGNCY6UCVS75CQSQRKIHGP37HHDI33AYVX2C5GG6XQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.977357831 (503h33m25.848440876s) [noise=true , address=77.90.31.19:4855 , piece_id=ANUQHCXR7SJ65GRILHY672CKW6B6LPPKNJ6EEBCEXB2B4YVVZDNA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:25.985224371 (503h33m25.856307426s) [noise=true , address=77.90.31.12:4601 , piece_id=IJSA33RP7ROMUEU2OTQTVFKXKNB2KCVIDMALMYP63VYVXMUEZYMA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.101992998 (503h33m25.973076033s) [noise=true , address=77.90.31.18:4810 , piece_id=O65ITTXPBIXSSHMKAUI3NQGIHNBF3UKG3DTZGS6RFPBPCUTOETPA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.348361146 (503h33m26.219444181s) [noise=true , address=45.91.116.5:4327 , piece_id=TTGSN4E3JARHYKHMBKKNXIY7LKZV6LWLPMAOUKVTJMHYYL2SVQAA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.569399721 (503h33m26.440482756s) [noise=true , address=77.90.31.17:4792 , piece_id=CNF5LCUK2HFGFOZUE6AZPYB2JHHWC7NUUWJCZ3AFSZO7P37DZESA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.572613080 (503h33m26.443696125s) [noise=true , address=145.79.227.40:4084 , piece_id=K6QCYEYGH5G5L7TM6BIBLFDVUMCABR6TZI2CAITG2CM2D3SWNGZQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.598950551 (503h33m26.470033596s) [noise=true , address=145.79.227.43:4161 , piece_id=EKRGUYX7B3GGGZB2RTIELHAXU67HB3ANMEXU6NKPORZ6V5ZLHGKA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.640277275 (503h33m26.51136032s) [noise=true , address=145.79.227.30:4776 , piece_id=UOC3L7XK6ZDYQ2HAW6MH3TDS34N4CF4ADB3BE6C6STKWMOIHH52A, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:26.673905639 (503h33m26.544988674s) [noise=true , address=145.79.227.44:4226 , piece_id=P3NGP5DEBXYJTNDME2VDPTTF47UBQHBQBYDBVSFRGO7FAOJVIGGQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201918006 (503h33m27.073001051s) [noise=true , address=64.49.13.170:4718 , piece_id=JCE6J55US2BRNJXSAQRVOFYY35T4AUKUJ23D5LKJI3BZHYI2CKWA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: <nil>
Dec 26 15:02:27.201937121 (503h33m27.073020156s) [seg_pos={0 0}, upload=441894]: Segment reached optimal threshold of 14 pieces.
Dec 26 15:02:27.202087527 (503h33m27.073170563s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202093450 (503h33m27.073176495s) [noise=true , address=45.91.116.10:4093 , piece_id=4373376DLCCPKC76REOSR7PWF6TYPKMCNST5TZHG3OZOZVBX54TQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202330550 (503h33m27.073413595s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202337284 (503h33m27.073420339s) [noise=true , address=77.90.31.11:4560 , piece_id=6J532TJPUDOC4DTZFTDNDRO2PVNFNYEWHOYC2MID7JUILLC4WDCQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202441242 (503h33m27.073524277s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF
Dec 26 15:02:27.202446502 (503h33m27.073529547s) [noise=true , address=77.90.31.13:4653 , piece_id=EADRWPSLXKRA6YQ3JJG5NSJDXU5GMZ3DOJSDOSPGW4Y2WIFL3WXA, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202551953 (503h33m27.073634998s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56938->64.49.13.169:4708: use of closed network connection
Dec 26 15:02:27.202557944 (503h33m27.073640989s) [noise=true , address=64.49.13.169:4708 , piece_id=PM5HRLQHTSZBDTYOZJI7UDCJ5JM4GALG4BE7NHYKVYLMM5OYSUJQ, seg_pos={0 0}, upload=441894]: Long tail cancellation due to optimal threshold
Dec 26 15:02:27.202571960 (503h33m27.073654995s) [seg_pos={0 0}, upload=441894]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 15:02:27.202628909 (503h33m27.073711954s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 15:02:27.202633117 (503h33m27.073716152s) [upload=441894]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 15:02:27.509671911 (503h33m27.380754946s) [upload=441894]: Done uploading segments...
Dec 26 15:02:27.509703091 (503h33m27.380786136s) [upload=441894]: Upload finished. err: <nil>
Duration: 54.794199481s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
15.440182274s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 15:06:05 UTC)
orbiter warning: context deadline exceeded
Duration: 15.440211048s
edge
gateway
us-select-1
|