Setup JsonFormatter to use logingstash format

[Timelessprod]

Hello,

I'm using your library and would need to set it up to output logs in the logstash format so they get parsed correctly by third party services using logstash. for example the current asctime and levelname don't get recognized.

I was tempted to use https://github.com/mbarrientos/logstash-python-formatter but it's not maintained for too long now and this may cause security problems.

I read the README and its section about adding extra fields but it seems to be outdated as I can'T see this fields in the class builder of JsonFormatter. Could you please guide me on what to do to change to rename fields and add new ones ?

Thanks!

[nhairs]

Hi @Timelessprod,

Just so you're aware it looks like python-json-logger is currently unmaintained, that said I am working on a maintained fork.

Whilst the below examples have been tested using my fork, I'm pretty sure that this still applies to the original library as well.

Renaming fields

Any field that you wish to rename that a standard LogRecord attribute needs to be included in the format of the Formatter.

import logging
# https://github.com/nhairs/python-json-logger v3.1.0.rc2
from pythonjsonlogger.json import JsonFormatter

## Setup
## -------------------------------------
logger = logging.getLogger("test")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
logger.addHandler(handler)

## Problem
## -----------------------------------------------------------------------------
formatter = JsonFormatter(
    rename_fields={"asctime": "logstash_asctime", "levelname": "logstash_levelname"}
)
handler.setFormatter(formatter)

logger.info("this does not bring me joy")

# --- Logging error ---
# Traceback (most recent call last):
#   File "/usr/lib/python3.10/logging/__init__.py", line 1100, in emit
#     msg = self.format(record)
#   File "/usr/lib/python3.10/logging/__init__.py", line 943, in format
#     return fmt.format(record)
#   File "/home/nhairs/git/self/python-json-logger/src/pythonjsonlogger/core.py", line 234, in format
#     self.add_fields(log_record, record, message_dict)
#   File "/home/nhairs/git/self/python-json-logger/src/pythonjsonlogger/core.py", line 312, in add_fields
#     self._perform_rename_log_fields(log_record)
#   File "/home/nhairs/git/self/python-json-logger/src/pythonjsonlogger/core.py", line 317, in _perform_rename_log_fields
#     log_record[new_field_name] = log_record[old_field_name]
# KeyError: 'asctime'
# Call stack:
#   File "/home/nhairs/git/scrap/python_json_logger_190.py", line 18, in <module>
#     logger.info("this does not bring me joy")
# Message: 'this does not bring me joy'
# Arguments: ()

## Solution
## -----------------------------------------------------------------------------
formatter = JsonFormatter(
    "{message}{asctime}{levelname}",
    style="{",
    rename_fields={"asctime": "logstash_asctime", "levelname": "logstash_levelname"}
)
handler.setFormatter(formatter)

logger.info("this brings me joy")

# {"message": "this brings me joy", "logstash_asctime": "2024-05-14 20:13:39,101", "logstash_levelname": "INFO"}

Note: that the asctime field format is controlled by the datefmt argument not the JSON encoder.

Adding fields

There's a few ways that fields can be added depending on your exact use case.

When making a logging call you can pass in the extra argument.

logging.info("some message", extra={"some_id": get_my_id()})

You can override the process_log_record method:

class MyFormatter(JsonFormatter):
    def process_log_record(log_record):
        log_record["some_id"] = get_my_id()
        return log_record

You can set static_fields:

formatter = JsonFormatter(static_fields={"service_name": "my_service"})

You can attach data to the LogRecord object using a filter, the additional LogRecord attributes will automatically be logged. https://docs.python.org/3/howto/logging-cookbook.html#using-filters-to-impart-contextual-information