How grafana determines profile type ？

[pyhita]

I sampled the heap memory information of a Java program through async-profiler and uploaded it to pyroscope through HTTP. The following is the uploaded code:

func main() {
	pyroscopeServer := "http://localhost:4040"
	applicationName := fmt.Sprintf("kante-test-test-sg{lang=java,env=test}")

	// Pprof file path
	pprofFilePath := fmt.Sprintf("/Users/kante.yang/pprof/java-propf")

	flameGraphData, err := fileutils.ReadFile(pprofFilePath)

	// Pyroscope URL
	pyroscopeURL := fmt.Sprintf("%s/ingest?name=%s", pyroscopeServer, applicationName)
	fmt.Println(applicationName)
	fmt.Println(pyroscopeURL)

	// Create the HTTP request
	req, err := createPythonJavaReq(flameGraphData, pyroscopeURL)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	// Send the request
	client := &http.Client{Timeout: 10 * time.Second}
	resp, err := client.Do(req)
	defer resp.Body.Close()
	if err != nil {
		fmt.Println("Error sending request:", err)
		return
	}

	// Check the response status
	if resp.StatusCode == http.StatusOK {
		fmt.Println("Profile data successfully sent to Pyroscope.")
	} else {
		fmt.Printf("Failed to send data. Status code: %d, Message: %s\n", resp.StatusCode, resp.Status)
	}
}

func createPythonJavaReq(rawProfileData []byte, url string) (*http.Request, error) {
	req, err := http.NewRequest("POST", url, bytes.NewBuffer(rawProfileData))
	if err != nil {
		return nil, errors.Wrap(err, "create pyroscope request error")
	}
	req.Header.Set("Content-Type", "application/octet-stream")

	return req, nil
}

I originally thought that pyroscope would automatically parse the format of the data. In fact, it did this for the data generated by pprof. However, after this data was uploaded to pyroscope, it was parsed into cpu, as shown below:

I'm not sure why this happens, or can I specify the profiletype when uploading data using the http api?

[pyhita]

I use this command to generate java profile data: profiler.sh -o collapsed -e allocs

[pyhita]

I found I can render memory flamegraph through specific application name: application.alloc_objects, application.insure_objects,application.insure_space,application.insure_space, but this means I need to initiate four http requests, and I also found insure_space and alloc_space have the same result, that's wired.Can anyone tell me how to initiate a request to reduce the number of http requests, and why the results of alloc and insure are the same?

[kolesnikovae]

Hi @pyhita! Could you please tell us more about your use case?

I'm not sure why this happens, or can I specify the profiletype when uploading data using the http api?

You can specify the profile type (multiple sample types, actually) via the sample_type_config part – please refer to ours docs

[pyhita]

Hi @kolesnikovae , Our usage scenarios for pyroscope are rather strange, because we do not continue profiling, but allow users to trigger profiling by clicking on the UI: if it is a Golang application, the profile file will be generated through the pprof tool after the user clicks it; if it is Java application will generate the profile file through the async tool; if it is a Python application, the profile file will be generated through pyflame.After generating the profile file, we will upload the file to pyroscope through the /ingest api, and then return a spliced ​​URL to the user. The user can see the flame graph generated by the profile in the pyroscope UI.We have too many application services. If each application uses an agent to upload profile information, the pressure will be too great. Therefore, our positioning is to allow users to manually trigger profiles when they need them.

After testing, I found that pyroscope has very good support for profile files generated by pprof. I do not need to specify the profile type, and it will automatically help me parse it. For example, if I upload the file generated by pprof -allocs, it will automatically help me parse it into alloc.inspace, alloc. .object, inuse.space, inuse.object, but for Java and Python profile files, they are collapsed format files. After I uploaded the Java memory profile file, pyroscope parsed it into cpu type.I have read the document you gave, but it seems to only allow pprof files to do this. For collapsed format files, I only know to set the applicationName to kante-test-test-sg.alloc_space, and then pyroscope will parse it into alloc_space. information, but this means I need to upload four times to get: alloc_space, alloc_object, inuse_space, inuse_object. I want to know how I can upload pyroscope for collapsed format files, will it be parsed into the correct format for me?

for collapsed format I refer this example:https://grafana.com/docs/pyroscope/latest/configure-server/about-server-api/#examples
for pprof format I refer this example: https://github.com/grafana/pyroscope/blob/main/examples/api/ingest_pprof.py

[kolesnikovae]

Hi @pyhita,

Thank you for the detailed response! I must say that your use case is absolutely valid and reasonable.

The collapsed format is quite limited and doesn’t support profiles with mixed sample types (like in pprof). I'd recommend converting the data to pprof in this case. As you noted, it's our primary format and highly flexible. You might find our JFR parser helpful.

I want to know how I can upload pyroscope for collapsed format files, will it be parsed into the correct format for me?

For profiles in "collapsed" format (and any other text-based format), the type should be specified explicitly in the URL query parameters

[pyhita]

Hi @pyhita,

Thank you for the detailed response! I must say that your use case is absolutely valid and reasonable.

The collapsed format is quite limited and doesn’t support profiles with mixed sample types (like in pprof). I'd recommend converting the data to pprof in this case. As you noted, it's our primary format and highly flexible. You might find our JFR parser helpful.

I want to know how I can upload pyroscope for collapsed format files, will it be parsed into the correct format for me?

For profiles in "collapsed" format (and any other text-based format), the type should be specified explicitly in the URL query parameters

@kolesnikovae Thank you for your information, this is very helpful to us!