victoriametrics.com

VictoriaLogs key concepts

Data model#

VictoriaLogs works with both structured and unstructured logs. Every log entry must contain at least log message field plus arbitrary number of additional key=value fields. A single log entry can be expressed as a single-level JSON object with string keys and values. For example:

{
  "job": "my-app",
  "instance": "host123:4567",
  "level": "error",
  "client_ip": "1.2.3.4",
  "trace_id": "1234-56789-abcdef",
  "_msg": "failed to serve the client request"
}

VictoriaLogs automatically transforms multi-level JSON (aka nested JSON) into single-level JSON during data ingestion according to the following rules:

  • Nested dictionaries are flattened by concatenating dictionary keys with . char. For example, the following multi-level JSON is transformed into the following single-level JSON:

    {
  "host": {
    "name": "foobar"
    "os": {
      "version": "1.2.3"
    }
  }
}

    {
  "host.name": "foobar",
  "host.os.version": "1.2.3"
}

  • Arrays, numbers and boolean values are converted into strings. This simplifies full-text search over such values. For example, the following JSON with an array, a number and a boolean value is converted into the following JSON with string values:

    {
  "tags": ["foo", "bar"],
  "offset": 12345,
  "is_error": false
}

    {
  "tags": "[\"foo\", \"bar\"]",
  "offset": "12345",
  "is_error": "false"
}

Both label name and label value may contain arbitrary chars. Such chars must be encoded during data ingestion according to JSON string encoding. Unicode chars must be encoded with UTF-8 encoding:

{
  "label with whitepsace": "value\nwith\nnewlines",
  "Поле": "价值",
}

VictoriaLogs automatically indexes all the fields in all the ingested logs. This enables full-text search across all the fields.

VictoriaLogs supports the following field types:

Message field#

Every ingested log entry must contain at least a _msg field with the actual log message. For example, this is the minimal log entry, which can be ingested into VictoriaLogs:

{
  "_msg": "some log message"
}

If the actual log message has other than _msg field name, then it is possible to specify the real log message field via _msg_field query arg during data ingestion. For example, if log message is located in the event.original field, then specify _msg_field=event.original query arg during data ingestion.

Time field#

The ingested log entries may contain _time field with the timestamp of the ingested log entry. For example:

{
  "_msg": "some log message",
  "_time": "2023-04-12T06:38:11.095Z"
}

If the actual timestamp has other than _time field name, then it is possible to specify the real timestamp field via _time_field query arg during data ingestion. For example, if timestamp is located in the event.created field, then specify _time_field=event.created query arg during data ingestion.

If _time field is missing, then the data ingestion time is used as log entry timestamp.

The log entry timestamp allows quickly narrowing down the search to a particular time range. See these docs for details.

Stream fields#

Some structured logging fields may uniquely identify the application instance, which generates log entries. This may be either a single field such as instance=host123:456 or a set of fields such as (datacenter=..., env=..., job=..., instance=...) or (kubernetes.namespace=..., kubernetes.node.name=..., kubernetes.pod.name=..., kubernetes.container.name=...).

Log entries received from a single application instance form a log stream in VictoriaLogs. VictoriaLogs optimizes storing and querying of individual log streams. This provides the following benefits:

  • Reduced disk space usage, since a log stream from a single application instance is usually compressed better than a mixed log stream from multiple distinct applications.

  • Increased query performance, since VictoriaLogs needs to scan lower amounts of data when searching by stream labels.

VictoriaLogs cannot determine automatically, which fields uniquely identify every log stream, so it stores all the received log entries in a single default stream - {}. This may lead to not-so-optimal resource usage and query performance.

Therefore it is recommended specifying stream-level fields via _stream_fields query arg during data ingestion. For example, if logs from Kubernetes containers have the following fields:

{
  "kubernetes.namespace": "some-namespace",
  "kubernetes.node.name": "some-node",
  "kubernetes.pod.name": "some-pod",
  "kubernetes.container.name": "some-container",
  "_msg": "some log message"
}

then sepcify _stream_fields=kubernetes.namespace,kubernetes.node.name,kubernetes.pod.name,kubernetes.container.name query arg during data ingestion in order to properly store per-container logs into distinct streams.

How to determine which fields must be associated with log streams?#

Log streams can be associated with fields, which simultaneously meet the following conditions:

  • Fields, which remain constant across log entries received from a single application instance.
  • Fields, which uniquely identify the application instance. For example, instance, host, container, etc.

Sometimes a single application instance may generate multiple log streams and store them into distinct log files. In this case it is OK to associate the log stream with filepath fields such as log.file.path additionally to instance-specific fields.

Structured logs may contain big number of fields, which do not change across log entries received from a single application instance. There is no need in associating all these fields with log stream - it is enough to associate only those fields, which uniquely identify the application instance across all the ingested logs. Additionally, some fields such as datacenter, environment, namespace, job or app, can be associated with log stream in order to optimize searching by these fields with stream filtering.

Never associate log streams with fields, which may change across log entries of the same application instance. See these docs for details.

High cardinality#

Some fields in the ingested logs may contain big number of unique values across log entries. For example, fields with names such as ip, user_id or trace_id tend to contain big number of unique values. VictoriaLogs works perfectly with such fields unless they are associated with log streams.

Never associate high-cardinality fields with log streams, since this may result to the following issues:

  • Performance degradation during data ingestion and querying
  • Increased memory usage
  • Increased CPU usage
  • Increased disk space usage
  • Increased disk read / write IO

VictoriaLogs exposes vl_streams_created_total metric, which shows the number of created streams since the last VictoriaLogs restart. If this metric grows at a rapid rate during long period of time, then there are high chances of high cardinality issues mentioned above. VictoriaLogs can log all the newly registered streams when -logNewStreams command-line flag is passed to it. This can help narrowing down and eliminating high-cardinality fields from log streams.

Other fields#

The rest of structured logging fields are optional. They can be used for simplifying and optimizing search queries. For example, it is usually faster to search over a dedicated trace_id field instead of searching for the trace_id inside long log message. E.g. the trace_id:XXXX-YYYY-ZZZZ query usually works faster than the _msg:"trace_id=XXXX-YYYY-ZZZZ" query.

See LogsQL docs for more details.

Hide

Back to Page