Node.js library for logging to LogDNA
- Migrating From Other Versions
- Install
- Setup
- Usage
- Default Log Levels
- Custom Log Levels
- Convenience Methods
- Events
- API
- How Log Lines are Sent to LogDNA
- Exponential Backoff Strategy
- Best Practices
- Client Side Support
- Bunyan Stream
- Winston Transport
- Using with AWS Lambda
- Proxy Support
- License
- Contributing
Previous versions of this client are still supported, but if you are upgrading to this version, please see our migration document for the differences between this version and prior versions.
$ npm install @logdna/logger
Operation requires a LogDNA Ingestion Key. Without it, the client will not be able to post logs to the cloud. Please contact our support if you have questions about this setup process!
To use the client, create an instance, then call .log()
or a convenience method.
const logdna = require('@logdna/logger')
const options = {
app: 'myAppName'
, level: 'debug' // set a default for when level is not provided in function calls
}
const logger = logdna.createLogger('<YOUR INGESTION KEY>', options)
logger.log('This is an INFO statement', 'info')
logger.log('This will be a DEBUG statement, based on the default')
logger.log('This is an INFO statement with an options object', {
level: 'info'
, meta: {
somekey: 'Arbitrary message'
, anotherkey: 'Another arbitrary message or data point'
}
})
logger.info('This is an INFO statement using a convenience method')
// Objects can be logged, too, but they're just serialized
logger.info({
message: 'Got some user data'
, userId: req.params.userId // This assumes `req.params` comes from some HTTP framework
})
// Just sets `level: 'error'` automatically
logger.error('An error was encountered while processing user data')
Although the logger can accept an object as its "message", Error
instances contain
non-enumerable properties such that log.error(err)
will not yield the expected results.
To mitigate this, users can trap (or generate) errors, then expose the error properties
as desired.
This example hardcodes some JSON to parse, but it could easily come from user input.
const input = '{"whoops": "This JSON is malformed because it\'s missing a closing quote}'
try {
JSON.parse(input)
} catch (err) {
log.error('JSON parse error while processing a string that should be JSON', {
indexMeta: true // Makes `meta` searchable. See docs below.
, meta: {
name: err.name
, message: err.message
, stack: err.stack
, input
}
})
// OR, if the all the details aren't important, a more concise log could be this
log.error(err.message, {
meta: {
message: 'JSON parse error during function xxx'
, input
}
})
}
The client supports the following log levels by default. They are case-insensitive. Users may also add custom log levels.
TRACE
DEBUG
INFO
WARN
ERROR
FATAL
Users may provide an array of levels
as a logger instantiation option. The levels
value must be an array, and its values must be letters only. All level values are
normalized to upper-case when sent to the LogDNA server, but their use in function
calls is case-insensitive.
const {createLogger} = require('@logdna/logger')
const logger = createLogger(myKey, {
levels: ['info', 'warn', 'critical', 'catastrophic']
})
logger.info('my text') // ok
logger.warn('some warning text') // ok
logger.catastrophic('OH NO!') // error
logger.log('OH NO!', 'critical') // ok
logger.log('We are crashing!', 'catastrophic') // ok
We have set up convenience methods that automatically set the log level appropriately, and are easy to read. If using custom log levels, then convenience methods will
only be added for custom levels that also match the default log levels, e.g. log.info()
.
msg
<Object>
|<String>
- The message (or object) to logoptions
<Object>
- Per-message options. Seelogger.log()
for those.
The Logger
is an EventEmitter
and will emit events rather than use promises or callbacks to communicate its progress.
Listening to the events is optional, although an error
listener is recommended.
Emitted when a meta property is successfully added. This meta property will be attached to each log message.
<Object>
message
<String>
- A message indicating that everything was sent or that nothing needed to be sent
When all log lines have been sent to LogDNA, this event is emitted. If it emits after lines have successfully been sent,
then the message will be 'All accumulated log entries have been sent'
. If there were no lines to be sent
(for example, if flush()
was called proactively), then the message will be 'All buffers clear; Nothing to send'
.
This event is emitted when an asynchronous error is encountered. Depending on the context, meta
will contain
different pieces of information about the error. If the error is retry-able, the error's message
property will indicate that it's a "temporary" error to avoid confusion with hard errors.
Note: ignoreRetryableErrors
is true
by default, and will not emit errors when
the retrying
property in the metadata is true
. To emit all errors regardless of
retrying
, set ignoreRetryableErrors: false
.
The metadata for an error encountered during an HTTP call to LogDNA will have the following meta
properties in the error.
actual
<String>
- The raw error message from the HTTP clientcode
<String>
|<Number>
- The HTTP agent's "code" orstatusCode
valuefirstLine
<String>
- The first log line in the buffer that was sendinglastLine
<String>
- The last log line in the buffer that was sendingretrying
<Boolean>
- Whether an attempt will be made to resend the payloadattempts
<Number>
- The number of consecutive failures
When log()
is called directly, or indirectly (via a convenience method), errors can be emitted if certain validations fail.
If an invalid log level is provided, or if a bad data type is found for the options
parameter, the meta
property of the error
will contain the following properties:
got
<String>
- Description of the invalid input. Will depend on error context.expected
<String>
- The allowable log levels ifoptions
is an invalid log levelused
<String>
- If a badlevel
is used inoptions
, it will be ignored, and the default will be used. This property indicates what that value is.
When log()
or agentLog()
is called, the payloadStructure
must be set appropriately. If it is not, an error is emitted.
Keep in mind that agentLog()
is reserved for LogDNA systems and is not intended for public usage.
message
<String>
- Static message ofInvalid method based on payloadStructure
payloadStructure
<String>
- The current payload structure value that is set on the instanceexpected
<String>
- The expected payload structure to be able to call the method.
If a 207
status code is received, this means that some of the lines failed to be ingested. An error
event is emitted for each
failed line and will have the following structure:
message
<String>
- Static message:Non-200 status while ingesting this line
meta
<Object>
- Details about the failed line
This is emitted when a key (and implied value) are removed from the global meta
object. If the key does not exist,
then a warn
event with the same signature will be emitted instead.
<Object>
httpStatus
<String>
- Thestatus
property of the HTTP agent's responsefirstLine
<String>
- The first log line in the buffer that was sentlastLine
<String>
- The last log line in the buffer that was senttotalLinesSent
<Number>
- The total number of lines in the sent buffertotalLinesReady
<Number>
- The number of lines left to be sent (if queueing has happened)bufferCount
<Number>
- The number of buffers left to be sent (if queueing has happened)
This event is emitted when a buffer is successfully sent to LogDNA. Since a buffer can contain many log entries, this event summarizes the activity.
In a high throughput system where flushLimit
is exceeded and multiple buffers are waiting to be sent, information
like totalLinesReady
and bufferCount
help illustrate how much work is left to be done. Any buffers that have been queued will
be sent one after another, ignoring any flush timer.
This event is emitted when there is no log data provided to the log
method, or when removeMetaProperty
is called with an unrecognized key.
For those cases, additional properties (apart from message
) are included:
statement
(Any) - Iflog()
was called with anull
string or an invalid data type, this key will contain the given log statement.
statement
(Any) - IfagentLog()
was called with anull
string or an invalid data type, this key will contain the given log statement.
key
<String>
- The key that the command attempted to remove but that did not exist
key
<String>
- Your ingestion keyoptions
<Object>
level
<String>
- Level to be used if not specified elsewhere. Default:INFO
levels
<Array>
- An array of custom log levels to use. Default: Default log levelstags
<Array>
|<String>
- Tags to be added to each messagemeta
<Object>
- Global metadata. Added to each message, unless overridden.timeout
<Number>
- Millisecond timeout for each HTTP request. Default:30000
ms. Max:300000
mshostname
<String>
- Hostname for each HTTP request.mac
<String>
- MAC address for each HTTP request.ip
<String>
- IPv4 or IPv6 address for each HTTP request.url
<String>
- URL of the logging server. Default:https://logs.logdna.com/logs/ingest
flushLimit
<Number>
- Maximum total line lengths before aflush
is forced. Default:5000000
flushIntervalMs
<Number>
- Mseconds to wait before sending the buffer. Default:250
msshimProperties
<Array>
- List of dynamicoptions
keys to look for when callinglog()
indexMeta
<Boolean>
- Controls whethermeta
data for each message is searchable. Default:false
app
<String>
- Arbitrary app name for labeling each message. Default:default
env
<String>
- An environment label attached to each messagebaseBackoffMs
<Number>
- Minimum exponential backoff time in milliseconds. Default:3000
msmaxBackoffMs
<Number>
- Maximum exponential backoff time in milliseconds. Default:30000
msmaxAttempts
<Number>
- Maximum number of times the logger will try to send a buffer of messages when retryable errors are encountered; when the limit is reached, retryable errors will be treated as non-retryable errors. Default:-1
, meaning unlimited.withCredentials
<Boolean>
- Passed to the request library to make CORS requests. Default:false
verboseEvents
<Boolean>
- Include the complete content of the buffer sent when emittingsend
anderror
events. When this option is enabled, the events will include an additionalbuffer
field which is an array of the messages and any metadata associated with those messages that were involved in the transmission that triggered the event. Default:false
payloadStructure
<String>
- (LogDNA usage only) Ability to specify a different payload structure for ingestion. Default:default
compress
<Boolean>
- (LogDNA usage only) Compression support for the agent. Default:false
proxy
<String>
- The full URL of an http or https proxy to pass throughignoreRetryableErrors
<Boolean>
- Do not emit "errors" that are retry-able. Typically, theses are temporary connection-based errors. Default:true
sendUserAgent
<Boolean>
- This option controls the sending of our library's user-agent string in HTTP requests to LogDNA. When this setting istrue
in a browser context, it may print a console error although the payloads are still sent. Setting this tofalse
in a browser context will retain theuser-agent
header of the browser. Default:true
- Throws:
<TypeError>
|<TypeError>
|<Error>
- Returns:
Logger
Returns a logging instance to use. flushLimit
and flushIntervalMs
control when the buffer is sent to LogDNA.
The flushIntervalMs
timer is only started after lines are logged, and the flushLimit
is a size approximation based on the summation
of .length
properties of each log line. If the buffer size exceeds flushLimit
, it will immediately send the buffer and ignore
the flushIntervalMs
timer. Otherwise, a timer will repeatedly flush the buffer every flushIntervalMs
milliseconds,
as long as the buffer contains log entries.
If indexMeta
is false
, then the metadata will still appear in LogDNA search, but the fields themselves will not be searchable.
If this option is true
, then meta objects will be parsed and searchable up to three levels deep. Any fields
deeper than three levels will be stringified and cannot be searched.
WARNING: When this option is true
, your metadata objects across all types of log messages MUST have consistent
types, or the metadata object may not be parsed properly!
shimProperties
can be used to set up keys to look for in the options
parameter of a log()
call. If the specified keys
are found in options
, their key-values will be included the top-level of the final logging payload send to LogDNA.
payloadStructure
is only for LogDNA's use in other parts of the system such as our logging agent.
It is not intended to be used by public consumers, and it should be left to the default value.
For more information on the backoff algorithm and the options for it, see the Exponential Backoff Strategy section.
The same as createLogger()
, except for that it creates a singleton that will be reused if called again.
Users can call this multiple times, and the client package will maintain (create and/or return) the singleton.
Note that only the first call will instantiate a new instance. Therefore, any successive calls will ignore the provided parameters.
const logdna = require('@logdna/logger')
const logger = logdna.setupDefaultLogger('<YOUR KEY HERE>')
const sameLogger = logdna.setupDefaultLogger()
This method is for use exclusively by LogDNA, and is not intended for public logging.
- Emits: error
key
<String>
- The meta property's keyvalue
<String>
|<Number>
|<Boolean>
|<Object>
|<Array>
- The meta property's value- Emits:
addMetaProperty
This method adds a key-value to the global metadata, which is added to each log entry upon calling log()
.
Although meta
can be set on instantiation, this method provides a way to update it on-the-fly.
If options.meta
is also used in a log()
call, the key-value pairs from the global meta
will be merged with
options.meta
, and those new pairs will take precedence over any matching keys in the global metadata.
// This will use `meta` to track logs from different modules
const logger = createLogger('<YOUR API KEY>', {
meta: {
module: 'main.js' // Global default
}
})
logger.debug('This is the main module') // Uses global meta
// ... elsewhere, in another file, perhaps
logger.info('I am in module1.js', {
meta: {module: __filename} // Overrides global meta
})
- Emits:
cleared
When flush
is called, any messages in the buffer are sent to LogDNA. It's not necessary to call this manually, although it is useful
to do so to ensure clean shutdown (see Best Practices). When log
is called, it automatically starts a timer
that will call flush
, but it is idempotent and can be called at any time.
If log lines exist in the current buffer, it is pushed onto a send queue, and a new buffer is created. The send queue is processed and uploaded to LogDNA.
If no work needs to be done, the cleared
event is immediately emitted.
statement
<String>
|<Object>
- Text or object of the log entry. Objects are serialized.options
<String>
|<Object>
- A string representing a level or an object with the following elements:level
<String>
- Desired level for the current message. Default:logger.level
app
<String>
- App name to use for the current message. Default:logger.app
env
<String>
- Environement name to use for the current message. Default:logger.env
timestamp
<Number>
- Epoch ms time to use for the current message. Must be within 24 hours. Default:Date.now()
context
<Object>
- Synonym formeta
, but mutually exclusive. Ignored ifmeta
exists.indexMeta
<Boolean>
- Allows for themeta
to be searchable in LogDNA. Default:logger.indexMeta
meta
<Object>
- Per-message meta data. Combined with key-values created withaddMetaProperty
- Emits: warn, error
Sends a string or object to LogDNA for storage. If the convenience methods are used, they call this function
under the hood, so the options are the same. The only difference is that level
is automatically set in the convenience methods.
Attempts to remove the given key from the global meta
data object. If the key is not found, warn
is emitted.
In default operation, when log
functions are called, the line is added to a buffer to cut down on HTTP traffic to the server.
The buffer is flushed every flushIntervalMs
milliseconds or if the log line lengths grow beyond flushLimit
.
When flush
fires (or is called manually), the current buffer is put onto a send queue, and a new buffer is started. The send queue begins
sending to LogDNA. It will continue to send without pausing or honoring flushIntervalMs
as long as there are buffers in the send queue.
When the send queue is empty, cleared
is emitted.
- If a
207
status code was received, this means that at least one line failed ingestion. Each offending line and its status code will be emitted as an error. - User-level errors (such as
400
) will not be retried because they most likely would never be successful (if the message is deemed invalid), anderror
events are emitted for these errors, also. - Connection-based errors or
500
-level response status codes will be retried using an exponential backoff strategy, but will also emiterror
events along the way.- Connection error codes that will retry:
ECONNABORTED
(timeout),ECONNRESET
,EADDRINUSE
,ECONNREFUSED
,EPIPE
,ENOTFOUND
,ENETUNREACH
- HTTP status codes that will retry:
500
,502
,503
,504
,521
,522
,524
- Connection error codes that will retry:
When HTTP failures happen, if they are deemed "retryable" (non-user errors), then the client will pause for a short time before trying to resend. The algorithm it implements is an exponential backoff with a "jitter" strategy that uses random numbers statistically to spread out the wait times to avoid flooding.
The settings for baseBackoffMs
and maxBackoffMs
are used in this algorithm and serve as the lower and upper boundaries for the wait time.
These types of errors are blocking since they are related to timeouts and server errors. Logs will continue to buffer as normal, and if the HTTP calls becomes successful, they will begin to send immediately, and without pause.
- The client is optimized for high throughput. Using a single instance is no problem, but multiple instances can be created with the same key if desired.
- Set up an
error
listener so that your app is aware of problems. Things like HTTP errors are emitted this way. - When shutting down your application, ensure all log entries are flushed by waiting for a final
cleared
event during shutdown:
const {createLogger} = require('@logdna/logger')
const {once} = require('events')
const process = require('process')
const logger = createLogger('This is not a real key and will cause an error')
logger.on('error', console.error)
function onSignal(signal) {
logger.warn(`received signal ${signal} shutting down`, {meta: {signal}})
shutdown().catch(() => {})
}
async function shutdown() {
await once(logger, 'cleared')
}
process.on('SIGTERM', onSignal)
process.on('SIGINT', onSignal)
// For running this as a standalone example, an error will be shown due
// to the key being invalid, but it shows the signal handler message
// attempting to be sent for ingestion.
setTimeout(() => {
process.kill(process.pid)
}, 1000)
For logging from a browser, we recommend our @logdna/browser package which is designed for that purpose.
For Bunyan Stream support, reference our logdna-bunyan module.
For Winston support, reference our logdna-winston module.
AWS Lambda relays stdout
and stderr
output from your function's code to CloudWatch,
but you can easily set up a Logger
instance as shown above to send logs to LogDNA instead.
If you have existing code that uses console.log
and console.error
statements, you can
also override these console
methods to send output to LogDNA without changing your code:
'use strict'
const {once} = require('events')
const {createLogger} = require('@logdna/logger')
const options = {
env: 'env'
, app: 'lambda-app'
, hostname: 'lambda-test'
}
const logger = createLogger('API KEY HERE', options)
// Override console methods to send logs to both LogDNA and stdout/stderr
const {
log: consoleLog
, error: consoleError
} = console
console.log = function(message, ...args) {
logger.log(message)
consoleLog(message, ...args)
}
console.error = function(message, ...args) {
logger.error(message)
consoleError(message, ...args)
}
exports.handler = async function handler(event, context) {
logger.on('error', consoleError)
// Your code here
console.log('Informational log')
console.log({
example: 'this is a sample object log'
})
console.error('Error log')
// Ensure logs have been flushed to LogDNA before finishing
await once(logger, 'cleared')
}
The logger supports proxying for situations such as corporate proxies that require traffic
to be passed through them before reaching the outside world. For such implementations,
use the proxy
instantiation option to set the full URL
of the proxy. It supports both http and https proxy URLs. Under the hood, the logger uses
the https-proxy-agent package for this.
In this example, an http proxy (with credentials) is passed through before reaching LogDNA's secure ingestion endpoint:
const {createLogger} = require('@logdna/logger')
const logger = createLogger(apiKey, {
proxy: 'http://username:[email protected]:12345'
, app: 'myapp'
})
logger.info('Happy logging through your proxy!')
Copyright Β© LogDNA, released under an MIT license. See the LICENSE file and https://opensource.org/licenses/MIT
Happy Logging!
This project is open-sourced, and accepts PRs from the public for bugs or feature enhancements. These are the guidelines for contributing:
- The project uses Commitlint and enforces Conventional Commit Standard. Please format your commits based on these guidelines.
- An issue must be opened in the repository for any bug, feature, or anything else that will have a PR
- The commit message must reference the issue with an acceptable action tag in the commit footer, e.g.
Fixes: #5
- The commit message must reference the issue with an acceptable action tag in the commit footer, e.g.
Thanks goes to these wonderful people (emoji key):
Darin Spivey π» π π§ |
relative π» |
Ryan Mottley π§ |
Alan Chen π» |
Mike Del Tito π |
Nhut Tran π» |
Eric Satterwhite π |
This project follows the all-contributors specification. Contributions of any kind welcome!