Skip to content

PD profiling failed ocassionally #1146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aylei opened this issue Jan 19, 2022 · 3 comments
Open

PD profiling failed ocassionally #1146

aylei opened this issue Jan 19, 2022 · 3 comments
Labels
area/backend priority/p1 Major functionality may not work type/bug
Milestone

Comments

@aylei
Copy link

aylei commented Jan 19, 2022

Bug Report

Please answer these questions before submitting your issue. Thanks!

What did you do?

Profile all instances through dashboard

What did you expect to see?
All profiles succeed

What did you see instead?
Some profiles to PD instances failed

What version of TiDB Dashboard are you using (./tidb-dashboard --version)?
v5.4.0

middle_img_v2_314c24e9-d495-478a-9607-df4524564a9g

error msg

dbaas v5.4.0 测试的时候发现 dashboard manual profiling 失败

failed to fetch and write to temp file: failed to fetch profile with *.proto format: error.pd.client_request_failed: Failed to send PD API request, cause: Get "https://db-pd-0.db-pd-peer.tidb1373933076657974810.svc:2379/pd/api/v1/debug/pprof/profile?seconds=30": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x05\x00\x10\x00\x00\x00\x03\x00\x00\x00\xfa\x00\x06\x00\x10\x01@\x00\x04\x00\x10\x00\x00"
https://staging.debug.tidbcloud.com/orgs/1369847559691367379  
@aylei aylei added the type/bug label Jan 19, 2022
@shhdgit
Copy link
Member

shhdgit commented Feb 23, 2022

According to golang/go#21336 (comment), I'm not sure if using the same tls.Config instance for the proxy and the dashboard client at https://github.com/tikv/pd/blob/master/pkg/dashboard/dashboard.go#L78-L88 would cause this problem.

PTAL, thanks~ @HunDunDM

@breezewish
Copy link
Member

We should add TLS into the test cases to make sure that our features works in TLS environments. @shhdgit Could you help set them up?

@shhdgit
Copy link
Member

shhdgit commented Feb 24, 2022

Yeah, sure. And it looks like this case needs to be tested on the pd side too.

@breezewish breezewish added this to the 5.1.0 milestone Mar 31, 2022
@breezewish breezewish added area/backend priority/p2 Some feature may not work priority/p1 Major functionality may not work and removed priority/p2 Some feature may not work labels Apr 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/backend priority/p1 Major functionality may not work type/bug
Projects
None yet
Development

No branches or pull requests

3 participants