proposal: Add inspect stage to loki.process

[tpaschalis]

Promtail's pipeline stages have been historically hard to debug; especially for pipelines that heavily depend on the extracted map of values, conditionally apply to only a subset of entries, or make use of more complex features such as templating. Flow's rendition of pipeline_stages in the loki.process component suffer from the same underlying issues.

I'd like to suggest that we introduce an inspect pass-through stage that does not alter incoming log entries or the shared extracted map.

The stage will make use of a LogQL stream selector and a sampling fraction to determine which log entries should be inspected. For entries selected for inspection, the stage would collect the log content, labels and the values in the shared extracted map and either log them, or append them to the parent loki.process component's DebugInfo.

So, given a user who's trying to understand what's going on in the following pipeline, they should be able to insert the inspect stage in any step of the process, reload the Flow config and be able to gain some understanding on how their processing stages work.

stage {
	regex {
		expression = "^(?s)(?P<time>\\S+?) (?P<stream>stdout|stderr) (?P<content>.*)$"
	}
}
stage {
	timestamp {
		source = "time"
		format = "RFC3339"
	}
}

/ *
stage {
        inspect {
                selector          = '{appLabel="mysql"} |~ ".*noisy error.*"'
                sampling_fraction = 0.1
        }
}
*/

stage {
	output {
		source = "content"
	}
}
stage {
	labels {
		values = { src = "stream" }
	}
}`

[rfratto]

The stage will make use of a LogQL stream selector and a sampling fraction to determine which log entries should be inspected. For entries selected for inspection, the stage would collect the log content, labels and the values in the shared extracted map and either log them, or append them to the parent loki.process component's DebugInfo.

👍 I'm in favor of this. What should the behavior be if a user has two inspect stages in the same component?

[rfratto]

Side-tangent: a few weeks ago I was considering the possibility of a component called debug.variables, where users can define a named list of variables that get exported:

logging {
  level = default.variables.default.loglevel 
}

debug.variables "default" {
  variable "loglevel" {
    default = "debug"
  }

  variable "logsinspect" {
    default = false 
  }
}

loki.process "default" {
  ...

  stage {
    inspect {
      enable = debug.variables.default.logsinspect 
      ...
    }
  }
}

Then, one could change the values of variables using the API:

curl -XPOST -d 'true' https://localhost:12345/components/debug.variables.default/logsinspect

The UI could show a special view for debug.variables which would allow changing the values for variables within the UI itself (and showing the current values).

I think a component like this provides some basic level of remote configuration that would be helpful specifically for debugging without having to update the config file.

[tpaschalis]

What should the behavior be if a user has two inspect stages in the same component?

One idea is to have the inspect block use an optional label; so when two inspect stages were on at the same time, they could report their separate output with the corresponding label. WDYT?

I was considering the possibility of a component called debug.variables, where users can define a named list of variables that get exported

I like it 👍 People can already do that today by making use of a local.file component, so I'm all for streamlining this and providing a way to do this from the UI.

One thing we'll have to consider is what types of values will be allowed, is that only basic River types? Would the following be valid?

logging {
  write_to = debug.variables.default.writeto
}

debug.variables "default" {
  variable "writeto" {
    default = "loki.stoud.default.receiver"
  }
}

$ curl -XPOST -d 'loki.write.grafanacloud.receiver' https://localhost:12345/components/debug.variables.default/writeto

[rfratto]

Yeah, I think only basic river types would be valid, and they'd have to use a boolean with an if statement (#2638) to switch where data gets sent at runtime.

But I also don't want to derail your issue with my proposal :) I'll create a separate issue for it today.

[rfratto]

Moving the debug.variables discussion to grafana/agent#2772 to avoid derailing this too much.

One idea is to have the inspect block use an optional label; so when two inspect stages were on at the same time, they could report their separate output with the corresponding label. WDYT?

Sure, SGTM 👍

[wildum]

Another option could be to use the stream debugging feature: grafana/agent#6045

[wildum]

I created an issue to add live debugging support for the loki.process component. It should be able to provide insights on every stage without having to modify the pipeline.
@tpaschalis what do you think? Should we close this proposal?