error calling MarshalJSON / unsupported value: NaN #479
Comments
|
I have tried adding these two HPA configurations: apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-prod-phpfpm
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: valinor-api-prod
minReplicas: 4
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: phpfpm_active_processes
targetAverageValue: 6
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: hpa-prod-external
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: valinor-api-prod
minReplicas: 4
maxReplicas: 10
metrics:
- type: External
external:
metric:
name: custom.googleapis.com|phpfpm_active_processes
target:
averageValue: 6
type: AverageValueAnd none of them seem to work as they should. The one using external type does however look a little more weird than the other. (Edit: Doesn't it actually look like it's correctly finding the custom metrics?) Here is the result: I know there are some kind of statistics coming into Google. Since I can see a phpfpm_active_processes in the metrics explorer, as a custom metric. It does however look a bit sketchy, since the active processes doesn't seem to distribute across the different pods. At least according to metrics explorer in google. But when I look directly at the /fpm-status and keep refreshing, then I can see the active processes are different on each pod in most cases. I was hoping I could scale based on active fpm processes, but if that is not the case. Will it be possible to scale based on request-per-second to each pod? Something like in the first example in this custom metrics adapter? Edit: But then the hpa using external type sometimes look like this. And that is what I find weird. |
|
I have the same logs |
If you are talking about the prometheus-to-sd image version, then as you can see in my issue post that I am already using Edit: Ahh shit sorry, I read it in the wrong order. I have been looking at yaml configs for too long. I will try it out, thank you. And I can not use the phpfpm_active_processes as a HPA metrics. I did also try to get the raw by running a command like the one you suggest. And as you also say, the items are empty and I figured that was the reason the metrics not working. Can you give me any insights in what you might have done differently in your setup? |
|
Here is my full yaml for the I think it looks pretty similar to yours. And I'm using kubernetes v1.24.1 |
|
It does look very similar. I am afraid though, that it is not only this container/adapter that is causing me trouble. I don't know if you are using the Google Custom Metrics stackdriver adapter? I had some trouble setting it up to begin with, but had some more success after installing it like this from this issue But I am getting errors like this: Are you using this custom metrics adapter or are you using another one? If you are using this one, are you also getting these errors? If not, how did you set that up? |
|
I have just tried setting it up again with the different version number. And my items is still empty unfortunately. As I mention above, I'm pretty sure it's the google stackdriver adapter that's causing me trouble now. I have set it up as I said and I am not getting 403 forbidden errors anymore, like I did in the beginning. But as the errors above suggest, then there are still something wrong with the adapter. It is getting When running the command: Is there a command to check all namespaces for that metric? Just for fun. I'm still new to k8s and trying to learn as much as I can. I have no idea at this point what can be causing issues other than the stackdriver. And I'm kinda over it at this point. |
|
I have a suspicion that there might be some issues when the My PHP FPM metrics were not being sent at all to Metrics Explorer but after stepping through the code here - k8s-stackdriver/prometheus-to-sd/translator/stackdriver.go Lines 54 to 55 in 0463e9b ts elements of the slice with the problematic Distribution values I could get the phpfpm metrics to work:
defer wg.Done()
var timeSeries []*v3.TimeSeries
for _, singleTimeSeries := range ts[begin:end] {
anyDistributionValuesFound := false
for _, point := range singleTimeSeries.Points {
if point.Value.DistributionValue != nil {
anyDistributionValuesFound = true
}
}
if !anyDistributionValuesFound {
timeSeries = append(timeSeries, singleTimeSeries)
}
}
req := &v3.CreateTimeSeriesRequest{TimeSeries: timeSeries}What I assume is happening is further down the line when this bulk This is by no mean a fix for the underlying problem but just an observation / quick fix I was able to put in place - I assume these issues with the |
Hello, I am facing an issue trying to set up the prometheus-to-sd container.
This is the container config in the deployment:
And this is the php-fpm exporter container:
I can see the fpm-metrics container working, since I have set up proxy pass in the nginx container to forward /metrics to the fpm-metrics container on port 9253 just as a test, and I can see some data when accessing it.
The logs from the prometheus-to-sd container after running for 15-30 minutes:
I have googled but have come up empty handed, and the closest issue I could find was from 2018-2019 and hasn't had any activity in a long time. So thought I would try here.
I apologize if this is the wrong place to ask or if it has been resolved I just hadn't found it.
Hope someone is able to help. Thanks in advance.
The text was updated successfully, but these errors were encountered: