victoriametrics.com

The following tip changes can be tested by building VictoriaLogs from the latest commit of VictoriaMetrics repository according to these docs

tip #

  • FEATURE: Datadog data ingestion: added -datadog.streamFields and -datadog.ignoreFields flags to configured default stream and ignore fields. Useful for Datadog serverless plugin, which doesn’t allow to provide extra headers of query args.

  • BUGFIX: Datadog data ingestion: accepts message field as both string and object type to fix compatibility with Datadog serverless extension, which sends logs data in format, which is not documented. See this issue.

  • BUGFIX: vlinsert: order of VL-Msg-Field values now defines a priority of these fields and it’s now obvious for a user which field will be picked if multiple msg_field values exist in a row.

v1.4.0 #

Released at 2024-12-22

  • FEATURE: stats pipe: allow non-numeric field values at median and quantile stats functions.

  • FEATURE: improve performance of stats pipe and top pipe by up to 2x when these pipes are applied to logs with millions of unique by (...) groups.

  • FEATURE: stats pipe: add count_uniq_hash function, which counts the number of unique value hashes. This number is usually a good approximation to the number of unique values, so the count_uniq_hash can be used as a faster alternative to count_uniq.

  • FEATURE: stats pipe: improve performance of count_uniq and uniq_values functions when they are applied to fields with big number of unique values.

  • FEATURE: facets pipe: add an ability to return log fields with the same values across all the selected logs by adding keep_const_fields option. Such log fields aren’t interesting in most cases, so they aren’t returned by default.

  • FEATURE: in filter: improve performance for in(<query>) when the <query> returns big number of values.

  • FEATURE: HTTP querying APIs: allow passing arbitrary LogsQL filters to extra_filters and extra_stream_filters query args. See these docs and this feature request for details.

  • FEATURE: Grafana Loki data ingestion: add support of Loki healthcheck /insert/ready endpoint. See this issue.

  • FEATURE: stream_context pipe: return an error as soon as too many logs and/or log streams are passed to this pipe. This prevents from excess resource usage by the stream_context pipe when it is improperly used. It is expected that the results of this pipe are investigated by humans, who cannot inspect surrounding logs for millions of the logs passed to stream_context. This change addresses this and this issues.

  • BUGFIX: syslog data ingestion: correctly parse rows with multiple consecutive spaces between fields. See this issue.

  • BUGFIX: web UI: fix cursor reset in query input field. See this issue.

  • BUGFIX: sort pipe: fix improper sorting of numeric fields in some cases.

  • BUGFIX: properly return an empty minimum value from min stats function.

v1.3.2 #

Released at 2024-12-09

  • FEATURE: collapse_nums pipe: add an ability to prettify some patterns across collapsed numbers. For example, <N>.<N>.<N>.<N> is replaced with <IP4> when executing collapse_nums prettify pipe.

  • BUGFIX: stream_context pipe: fix index out of range [0] with length 0 panic, which has been introduced in v1.3.0-victorialogs. See this issue.

v1.3.1 #

Released at 2024-12-08

v1.3.0 #

Released at 2024-12-08

  • FEATURE: add collapse_nums pipe, which replaces all the decimal and hexadecimal numbers with <N> in the given log field. This can be useful for locating the most frequently seen log message patterns if log messages differ only by decimal and hexadecimal numbers (this is very frequent case). For example, the following query returns top 5 log message patterns seen over the last hour: _time:1h | collapse_nums | top 5 by (_msg).
  • FEATURE: improve performance for stream_context pipe over log streams with big number of logs (millions and more). See this issue.
  • FEATURE: stream_context pipe allow changing the time window for search for surrounding logs via time_window option. For example, the following query searches for surrouning log stream logs on the one week window: _time:5m error | stream_context before 10 time_window 1w. Thanks to @worker24h for the idea.

v1.2.0 #

Released at 2024-12-06

  • FEATURE: add rate and rate_sum stats functions, which can be used for calculating the average per-second rate of matching logs and the average per-second rate of sum over the given numberic log fields. See this feature request.

  • FEATURE: add facets pipe, which can be used for returning the most frequent values across all the fields seen in the selected logs. This pipe simplifies logs’ exploration.

  • FEATURE: add /select/logsql/facets HTTP endpoint, which returns the most frequent values across all the fields seen in the selected logs. This endpoint is going to be used for building faceted search over logs in the VictoriaLogs web UI.

  • BUGFIX: /select/logsql/stats_query: properly apply limit at stats pipe. The bug was introduced in the release v1.1.0 when fixing this issue.

  • BUGFIX: math pipe: properly format expressions with multiple binary operations with the same priority. For example, x / (y * z) was improperly formatted as x / y * z, while x - (y + z) was improperly formatted as x - y + z. This could lead to incorrect query results in vlogscli.

v1.1.0 #

Released at 2024-12-05

  • FEATURE: add first and last pipes for returning the first N and the last N logs after sorting them by the given set of log fields. For example, the following query returns up to 5 logs with the biggest value for request_duration over the last hour: _time:1h | last 5 by (request_duration).

  • FEATURE: sort pipe: add an ability to apply limit individually per group of logs via partition by (...) syntax. For example, the following query returns up to 3 logs with the smallest request_duration individually per each host: _time:5m | sort by (request_duration) limit 3 partition by (host).

  • FEATURE: format pipe: allow formatting log fields in lowercase and uppercase via <uc:field_name> and <lc:field_name> syntax. This can be useful when some fields must be consistently transformed to the same case during querying. See this issue.

  • FEATURE: web UI: add frontend-only pagination for table view.

  • FEATURE: web UI: improve memory consumption during data processing. This enhancement reduces the overall memory footprint, leading to better performance and stability.

  • FEATURE: web UI: reduce memory usage across all tabs for improved performance and stability. See this issue.

  • FEATURE: Grafana Loki data ingestion: use Loki stream labels as VictoriaLogs stream fields by default. The set of stream fields can be overriden via _stream_fields query arg or via VL-Stream-Fields header as described here.

  • FEATURE: OpenTelemetry data ingestion: use resource labels as VictoriaLogs stream fields by default. The set of stream fields can be overriden via _stream_fields query arg or via VL-Stream-Fields header as described here.

  • FEATURE: data ingestion: expose vl_bytes_ingested_total counter at /metrics page. This counter tracks an estimated number of bytes processed when parsing the ingested logs. This counter is exposed individually per every supported data ingestion protocol - the protocol name is exposed in the type label. For example, vl_bytes_ingested_total{type="jsonline"} tracks an estimated number of bytes processed when reading the ingested logs via json line protocol. Thanks to @tenmozes for the idea and the initial implementation.

  • FEATURE: data ingestion: expose vl_too_long_lines_skipped_total counter at /metrics page. This counter tracks the number of the ingested lines with the length bigger than the value of -insert.maxLineSizeBytes command-line flag. Such lines are ignored.

  • BUGFIX: /select/logsql/stats_query_range API: properly handle limit pipe after sort pipe. Previously the limit was applied globally across all the calculated stats, while it must be applied individually per each step on the start ... end time range. See this issue.

  • BUGFIX: vmui: fix for showLegend and alias flags in predefined panels. See this issue

  • BUGFIX: fix too big number of columns detected in the block panic when the ingested logs contain more than 2000 fields with different names per every log stream. See this issue for details.

  • BUGFIX: properly parse lines after too long JSON lines and Elasticsearch lines with the length exceeding -insert.maxLineSizeBytes. Previously all the lines after the too long line in the stream were ignored.

v1.0.0 #

Released at 2024-11-12

This release is identical to v0.42.0.

VictoriaLogs gained all the planned features since the initial v0.1.0 release 1.5 years ago, and is ready for production!

v0.42.0 #

Released at 2024-11-08

v0.41.0 #

Released at 2024-11-06

  • FEATURE: support structured metadata when ingesting logs with Grafana Loki ingestion protocol. See this issue.

  • FEATURE: add join pipe, which can be used for performing SQL-like joins.

  • FEATURE: support returning historical logs from live tailing API via start_offset query arg. For example, request to /select/logsql/tail?query=*&start_offset=5m returns logs for the last 5 minutes before starting returning live tailing logs for the given query.

  • FEATURE: add an ability to specify extra fields for logs ingested via HTTP-based data ingestion protocols. See extra_fields query arg and VL-Extra-Fields HTTP header in these docs.

  • FEATURE: add block_stats pipe for returning various per-block stats. This pipe is useful for debugging.

  • FEATURE: web UI: add sorting of logs by groups and within each group by time in desc order. See this issue and this issue.

  • FEATURE: add support for receiving DataDog logs over network. See this issue.

  • BUGFIX: properly sort fields with floating-point numbers by sort pipe. Previously floating-point numbers could be improperly sorted because they were treated as strings, and natural sorting was incorrectly applied to them. For example, 0.123 was treated as bigger than 0.9.

v0.40.0 #

Released at 2024-10-31

v0.39.0 #

Released at 2024-10-30

  • FEATURE: allow specifying a list of log fields, which may contain log message, via _msg_field query arg and via VL-Msg-Field HTTP request header. For example, _msg_field=message,event.message instructs obtaining message field from the first non-empty field out of the message and event.message fields. See these docs for details.

  • FEATURE: accept logs without _msg field. In this case the _msg field is automatically set to the value specified in the -defaultMsgValue command-line flag.

  • BUGFIX: fix runtime error: index out of range [0] with length 0 panic during low-rate data ingestion. The panic has been introduced in v0.38.0. See this issue.

v0.38.0 #

Released at 2024-10-29

  • FEATURE: added the ability to receive systemd (journald) logs over network. See this issue.

  • FEATURE: improve performance for queries over large volume of logs with big number of fields (aka wide events).

  • FEATURE: improve performance for /select/logsql/field_values HTTP endpoint.

  • FEATURE: improve performance for field_values pipe when it is applied directly to log filter.

  • FEATURE: add an ability to return rank field from top pipe. For example, the following query returns 1..5 rank per each returned ip with the biggest number of logs over the last 5 minute: _time:5m | top 5 by (ip) rank.

  • BUGFIX: web UI: fix various glitches with updating query responses. The issue was introduced in v0.36.0. See this issue.

v0.37.0 #

Released at 2024-10-18

  • FEATURE: web UI: add ability to hide hits chart. See this issue.

  • FEATURE: add basic alerting rules for VictoriaLogs process. See details at monitoring docs.

  • FEATURE: improve stats pipe performance on systems with many CPU cores when by(...) fields contain big number of unique values. For example, _time:1d | stats by (user_id) count() x should be executed much faster when user_id field contains millions of unique values.

  • FEATURE: improve performance for top, uniq and field_values pipes on systems with many CPU cores when it is applied to log fields with big number of unique values. For example, _time:1d | top 5 (user_id) should be executed much faster when user_id field contains millions of unique values.

  • FEATURE: improve performance for field_names pipe when it is applied to logs with hundreds of log fields.

  • BUGFIX: web UI: fix display of hits chart. See this issue.

v0.36.0 #

Released at 2024-10-16

  • FEATURE: optimize LogsQL queries, which need to scan big number of logs with big number of log fields (aka wide events). The performance for such queries is improved by 10x and more depending on the number of log fields in the scanned logs. The performance improvement is visible when querying logs ingested after the upgrade to this release.

  • FEATURE: add support for forced merge. See these docs.

  • FEATURE: skip empty log fields in query results, since they are treated as non-existing fields in VictoriaLogs data model. This should reduce the level of confusion for end users when they see empty log fields.

  • FEATURE: allow using format pipe for creating output labels from existing log fields at /select/logsql/stats_query and /select/logsql/stats_query_range endpoints.

  • FEATURE: web UI: add the ability to cancel running queries. See this issue.

  • BUGFIX: avoid possible panic when logs for a new day are ingested during execution of concurrent queries.

  • BUGFIX: avoid panic at lib/logstorage.(*blockResultColumn).forEachDictValue() when the query contains stats with additional filters. The panic has been introduced in v0.33.0 in this commit.

  • BUGFIX: add more checks for stats query APIs to avoid invalid results.

  • BUGFIX: vmui: fix error messages rendering from overflowing the screen with long messages. See this issue.

v0.35.0 #

Released at 2024-10-09

  • FEATURE: vlogscli: add ability to live tail query results - see these docs.
  • FEATURE: vlogscli: add compact output mode for query results. It can be enabled by typing \c and then pressing enter. See these docs.
  • FEATURE: vlogscli: add -accountID and -projectID command-line flags for setting AccountID and ProjectID values when querying the specific tenants.

v0.34.0 #

Released at 2024-10-08

  • FEATURE: vlogscli: add ability to display results in logfmt mode, single-line and multi-line JSON modes according these docs.
  • FEATURE: vlogscli: preserve less output after the exit from scrolling mode. This should help re-using previous query results in subsequent queries.
  • FEATURE: add len pipe for calculating the length for the given log field value in bytes.

v0.33.0 #

Released at 2024-10-01

v0.32.1 #

Released at 2024-09-30

v0.32.0 #

Released at 2024-09-29

  • FEATURE: data ingestion: accept Unix timestamps in seconds in the ingested logs. This simplifies integration with systems, which prefer Unix timestamps over text-based representation of time.

  • FEATURE: sort pipe: allow using order alias instead of sort. For example, _time:5s | order by (_time) query works the same as _time:5s | sort by (_time). This simplifies the to LogsQL transition from SQL-like query languages.

  • FEATURE: stats pipe: allow using multiple identical stats functions with distinct filters and automatically generated result names. For example, _time:5m | count(), count() if (error) query works as expected now, e.g. it returns two results over the last 5 minutes: the total number of logs and the number of logs with error word. Previously this query couldn’t be executed because the if (...) condition wasn’t included in the automatically generate result name, so both results had the same name - count(*).

  • BUGFIX: properly calculate uniq and top pipes. Previously they could return invalid results in some cases.

v0.31.0 #

Released at 2024-09-27

  • FEATURE: web UI: improved readability of staircase graphs and tooltip usability. See this comment.
  • FEATURE: web UI: simplify query input by adding only the label name when ctrl+clicking the line legend. See this comment.
  • FEATURE: web UI: keep selected columns in table view on page reloads. Before, selected columns were reset on each update. See this issue.
  • FEATURE: allow skipping _stream: prefix in stream filters. This simplifies writing queries with stream filters. Now {foo="bar"} is the recommended format for stream filters over the _stream:{foo="bar"} format.
  • FEATURE: allow using - instead of ! as NOT operator shorthand in logical filters. For example, -info -warn query is equivalent to !info !warn. This simplifies transition from other query languages with full-text search support, which usually use - as NOT operator.

v0.30.1 #

Released at 2024-09-27

  • BUGFIX: consistently return matching log streams sorted by time from stream_context pipe. Previously log streams could be returned in arbitrary order with every request. This could complicate using stream_context pipe.
  • BUGFIX: stream_context pipe: add missing _msg="---" delimiter between stream contexts belonging to different log streams. This should simplify investigating stream_context output for multiple matching log streams.

v0.30.0 #

Released at 2024-09-27

  • FEATURE: web UI: add button for enabling auto refresh, similarly to VictoriaMetrics vmui. See this issue.

  • FEATURE: drop logs without _msg field or with empty _msg field, since this field is required to be non-empty in VictoriaLogs data model. See this issue.

  • FEATURE: improve performance of analytical queries, which do not need reading the _time field. See this issue.

  • FEATURE: add blocks_count pipe, which can be used for counting the number of matching blocks for the given query. For example, _time:5m | blocks_count returns the number of blocks with logs for the last 5 minutes. This pipe can be useful for debugging purposes.

  • FEATURE: support ingesting logs with _time field, which doesn’t contain timezone information. For example, 2024-09-20T10:20:30. In this case the local timezone of the host where VictoriaLogs runs is used. See this issue.

  • FEATURE: reduce memory usage when stream_context pipe is applied to log streams with big number of messages. See this issue.

  • BUGFIX: fix Windows build, which has been broken in v0.29.0. See this issue.

  • BUGFIX: properly return logs from /select/logsql/tail endpoint if the query contains _time:some_duration filter like _time:5m. See this issue. The bug has been introduced in v0.29.0.

  • BUGFIX: properly return logs without _msg field when * query is passed to /select/logsql/query endpoint together with positive limit arg. See this issue. Thanks to @jiekun for identifying the root cause of the issue.

  • BUGFIX: support ingesting logs with _time field containing whitespace delimiter between the date and time instead of T delimiter. For example, 2024-09-20 10:20:30. This is valid ISO8601 format aka SQL datetime format, which sometimes is used in production. See this issue.

  • BUGFIX: return all the requested surrounding logs for stream_context pipe. Previously only logs matching the _time filter were returned. This is needed for this feature.

v0.29.0 #

Released at 2024-09-08

  • FEATURE: add /select/logsql/stats_query HTTP API, which is going to be used by vmalert for executing alerting and recording rules against VictoriaLogs. See this feature request for details.

  • FEATURE: add /select/logsql/stats_query_range HTTP API, which is going to be used by VictoriaLogs plugin for Grafana for building time series panels. See this feature request for details.

  • FEATURE: optimize multi-exact queries with many phrases to search. For example, ip:in(path:="/foo/bar" | keep ip) when there are many unique values for ip field among log entries with /foo/bar path.

  • FEATURE: web UI: add support for displaying the top 5 log streams in the hits graph. The remaining log streams are grouped into an “other” label. See this issue.

  • FEATURE: web UI: add the ability to customize the graph display with options for bar, line, stepped line, and points.

  • FEATURE: web UI: add fields for setting AccountID and ProjectID. See this issue.

  • FEATURE: web UI: add a toggle button to the “Group” tab that allows users to expand or collapse all groups at once.

  • FEATURE: web UI: introduce the ability to select a key for grouping logs within the “Group” tab.

  • FEATURE: web UI: display the number of entries within each log group.

  • FEATURE: web UI: move the Markdown toggle to the general settings panel in the upper left corner.

  • FEATURE: web UI: add search functionality to the column display settings in the table. See this issue.

  • FEATURE: web UI: add the ability to select all columns in the column display settings of the table. See this issue. Thanks to @yincongcyincong for the pull request.

  • FEATURE: Allow to define ingestion parameters via headers. Supported headers - VL-Msg-Field,VL-Stream-Fields,VL-Ignore-Fields,VL-Time-Field, VL-Debug. See this PR for details.

  • FEATURE: vlinsert: added OpenTelemetry logs ingestion support. See this PR for details.

  • BUGFIX: properly handle Logstash requests for Elasticsearch configuration when using outputs.elasticsearch in Logstash pipelines. Previously, the requests could be rejected with 400 Bad Request response. Updates this issue.

  • BUGFIX: vmui: fix not found index.js error when loading vmui in VictoriaLogs. See this issue. Thanks to @yincongcyincong for the pull request.

  • BUGFIX: properly execute queries with OR filters for distinct log fields. For example, field1:foo OR field2:bar. Previously logs matching these filters may be skipped during querying. See this issue for details. Thanks to @yincongcyincong for the pull request.

v0.28.0 #

Released at 2024-07-10

  • FEATURE: web UI: show a spinner on top of bar chart until user’s request is finished. See this issue.
  • FEATURE: web UI: use compact representation of JSON lines at JSON tab if only a single log field is queried. See this feature request.
  • FEATURE: web UI: properly show the number of matching logs on the selected time range at bar chart for queries with arbitrary pipes, including stats pipe and top pipe.

v0.27.1 #

Released at 2024-07-05

v0.27.0 #

Released at 2024-07-02

  • FEATURE: add -syslog.useLocalTimestamp.tcp and -syslog.useLocalTimestamp.udp command-line flags, which could be used for using the local timestamp as _time field for the logs ingested via the corresponding -syslog.listenAddr.tcp / -syslog.listenAddr.udp. By default the timestamp from the syslog message is used as _time field. See these docs.

  • BUGFIX: make slowly ingested logs visible for search as soon as they are ingested into VictoriaLogs. Previously slowly ingested logs could remain invisible for search for long time.

v0.26.1 #

Released at 2024-07-01

  • BUGFIX: return the proper surrounding logs for stream_context pipe when additional pipes are put after the stream_context pipe. This has been broken in v0.26.0.

v0.26.0 #

Released at 2024-07-01

  • FEATURE: add ability to return log position (aka rank) after sorting logs with sort pipe. This can be done by adding rank as <fieldName> to the end of | sort ... pipe. For example, _time:5m | sort by (_time) rank as position instructs storing position of every sorted log line into position field name.
  • FEATURE: add delimiter log with --- message between log chunks returned by stream_context pipe. This should simplify investigation of the returned logs.
  • FEATURE: reduce memory usage when big number of context logs are requested from stream_context pipe.

v0.25.0 #

Released at 2024-06-28

  • FEATURE: add ability to select surrounding logs in front and after the selected logs via stream_context pipe. This functionality may be useful for investigating stacktraces, panics or some correlated log messages. This functionality is similar to grep -A and grep -B.

  • FEATURE: add ability to return top N "fields" groups from /select/logsql/hits HTTP endpoint, by specifying fields_limit=N query arg. This query arg is going to be used in this feature request.

  • BUGFIX: fix runtime error: index out of range [0] with length 0 panic when empty lines are ingested via Syslog format by Cisco controllers. See this issue.

v0.24.0 #

Released at 2024-06-27

  • FEATURE: add /select/logsql/tail HTTP endpoint, which can be used for live tailing of LogsQL query results. See these docs for details.

  • FEATURE: add /select/logsql/stream_ids HTTP endpoint, which can be used for returning _stream_id values with the number of hits for the given LogsQL query. See these docs for details.

  • FEATURE: add -retention.maxDiskSpaceUsageBytes command-line flag, which allows limiting disk space usage for VictoriaLogs data by automatic dropping the oldest per-day partitions if the storage disk space usage becomes bigger than the -retention.maxDiskSpaceUsageBytes. See these docs.

  • BUGFIX: properly take into account query timeout specified via -search.maxQueryDuration command-line flag and/or via timeout query arg. Previously these timeouts could be ignored during query execution.

  • BUGFIX: web UI: fix the update of the relative time range when Execute Query is clicked. See this issue.

v0.23.0 #

Released at 2024-06-25

  • FEATURE: syslog data ingestion: parse STRUCTURED-DATA into SD-ID.field1=value1, SD-ID.field2=value2, …, SD-ID.fieldN=valueN log fields. Previously the STRUCTURED-DATA was parsed into a single log field with the SD-ID name and field1=value1 field2=value2 ... fieldN=valueN value. This could complicate querying of such data.

  • BUGFIX: properly parse timestamps with timezones during data ingestion and querying. This has been broken in v0.20.0. See this issue.

v0.22.0 #

Released at 2024-06-24

  • FEATURE: allow specifying multiple _stream_id values in _stream_id filter via _stream_id:in(id1, ..., idN) syntax.
  • FEATURE: allow specifying subquery for searching for _stream_id values inside _stream_id filter. For example, _stream_id:in(_time:5m error | fields _stream_id) returns logs for logs streams with the error word across logs for the last 5 minutes.

v0.21.0 #

Released at 2024-06-20

  • FEATURE: web UI: add a bar chart displaying the number of log entries over a time range. See this issue.
  • FEATURE: expose _stream_id field, which uniquely identifies log streams. This field can be used for quick obtaining of all the logs belonging to a particular stream via _stream_id filter.

v0.20.2 #

Released at 2024-06-18

v0.20.1 #

Released at 2024-06-18

v0.20.0 #

Released at 2024-06-17

  • FEATURE: add ability to accept logs in Syslog format. See these docs.
  • FEATURE: add ability to specify timezone offset when parsing rfc3164 syslog messages with unpack_syslog pipe.
  • FEATURE: add top pipe for returning top N sets of the given fields with the maximum number of matching log entries.

v0.19.0 #

Released at 2024-06-11

  • FEATURE: do not allow starting the filter with pipe names and stats function names. This prevents from unexpected results returned by incorrect queries, which miss mandatory filter.

  • FEATURE: treat unexpected syslog message as RFC3164 containing only the message field when using unpack_syslog pipe.

  • FEATURE: allow using where prefix instead of filter prefix in filter pipe.

  • FEATURE: disallow unescaped ! char in LogsQL queries, since it permits writing incorrect query, which may look like correct one. For example, foo!:bar instead of foo:!bar.

  • FEATURE: web UI: add markdown support to the Group view. See this pull request.

  • BUGFIX: return back the improved performance for queries with * filters (aka SELECT *). This has been broken in v0.16.0.

v0.18.0 #

Released at 2024-06-06

v0.17.0 #

Released at 2024-06-05

  • FEATURE: add pack_logfmt pipe for formatting log fields into logfmt messages.

  • FEATURE: allow using IPv4 addresses in range comparison filters. For example, ip:>'12.34.56.78' is valid filter now.

  • FEATURE: add ceil() and floor() functions to math pipe.

  • FEATURE: add support for bitwise and, or and xor operations at math pipe.

  • FEATURE: add support for automatic conversion of RFC3339 time and IPv4 addresses into numeric representation at math pipe.

  • FEATURE: add ability to format numeric fields into string representation of time, duration and IPv4 with format pipe.

  • FEATURE: set format field to rfc3164 or rfc5424 depending on the Syslog format parsed via unpack_syslog pipe.

  • BUGFIX: always respect the limit set in limit pipe. Previously the limit could be exceeded in some cases.

v0.16.0 #

Released at 2024-06-04

v0.15.0 #

Released at 2024-05-30

  • FEATURE: add row_any function for stats pipe. This function returns a sample log entry per every calculated group of results.

  • FEATURE: add default operator to math pipe. It allows overriding NaN results with the given default value.

  • FEATURE: add exp() and ln() functions to math pipe.

  • FEATURE: allow omitting result name in math pipe expressions. In this case the result name is automatically set to string representation of the corresponding math expression. For example, _time:5m | math duration / 1000 is equivalent to _time:5m | math (duration / 1000) as "duration / 1000".

  • FEATURE: allow omitting result name in stats pipe. In this case the result name is automatically set to string representation of the corresponding stats function expression. For example, _time:5m | count(*) is valid LogsQL query now. It is equivalent to _time:5m | stats count(*) as "count(*)".

  • BUGFIX: properly calculate the number of matching rows in * | field_values x | stats count() rows and in * | unroll (x) | stats count() rows queries.

v0.14.0 #

Released at 2024-05-29

  • FEATURE: allow specifying fields, which must be packed into JSON in pack_json pipe via pack_json fields (field1, ..., fieldN) syntax.

  • BUGFIX: properly apply if (...) filters to calculated results in stats pipe when grouping by fields is enabled. For example, _time:5m | stats by (host) count() logs, count() if (error) errors now properly calculates per-host errors.

v0.13.0 #

Released at 2024-05-28

  • FEATURE: add extract_regexp pipe for extracting arbitrary substrings from log fields with RE2 regular expressions.
  • FEATURE: add math pipe for mathematical calculations over log fields.
  • FEATURE: add field_values pipe, which returns unique values for the given log field.
  • FEATURE: allow omitting stats prefix in stats pipe. For example, _time:5m | count() rows is a valid query now. It is equivalent to _time:5m | stats count() as rows.
  • FEATURE: allow omitting filter prefix in filter pipe if the filter doesn’t clash with pipe names. For example, _time:5m | stats by (host) count() rows | rows:>1000 is a valid query now. It is equivalent to _time:5m | stats by (host) count() rows | filter rows:>1000.
  • FEATURE: allow head pipe without number. For example, error | head. In this case 10 first values are returned as head Unix command does by default.
  • FEATURE: allow using comparison filters with strings. For example, some_text_field:>="foo" matches log entries with some_text_field field values bigger or equal to foo.

v0.12.1 #

Released at 2024-05-26

  • FEATURE: add support for comments in multi-line LogsQL queries. See these docs.

  • BUGFIX: properly apply in(...) filter inside if (...) conditions at various pipes. This bug has been introduced in v0.12.0.

v0.12.0 #

Released at 2024-05-26

  • FEATURE: add pack_json pipe, which packs all the log fields into a JSON object and stores it into the given field.

  • FEATURE: add unroll pipe, which can be used for unrolling JSON arrays stored in log fields.

  • FEATURE: add replace_regexp pipe, which allows updating log fields with regular expressions.

  • FEATURE: improve performance for format and extract pipes.

  • FEATURE: improve performance for /select/logsql/field_names HTTP API.

  • BUGFIX: prevent from panic in sort pipe when VictoriaLogs runs on a system with one CPU core.

  • BUGFIX: do not return referenced fields if they weren’t present in the original logs. For example, _time:5m | format if (non_existing_field:"") "abc" could return empty non_exiting_field, while it shouldn’t be returned because it is missing in the original logs.

  • BUGFIX: properly initialize values for in(...) filter inside filter pipe if the in(...) contains other filters. For example, _time:5m | filter ip:in(user_type:admin | fields ip) now works correctly.

v0.11.0 #

Released at 2024-05-25

v0.10.0 #

Released at 2024-05-24

  • FEATURE: return the number of matching log entries per returned value in HTTP API results. This simplifies detecting field / stream values with the biggest number of logs for the given LogsQL query.

  • FEATURE: improve performance for regexp filter in the following cases:

    • If the regexp contains just a phrase without special regular expression chars. For example, ~"foo".
    • If the regexp starts with .* or ends with .*. For example, ~".*foo.*".
    • If the regexp contains multiple strings delimited by |. For example, ~"foo|bar|baz".
    • If the regexp contains multiple words. For example, ~"foo bar baz".

  • FEATURE: allow disabling automatic unquoting of the matched placeholders in extract pipe. See these docs.

  • BUGFIX: properly parse ! in front of exact filter, exact-prefix filter and regexp filter. For example, !~"some regexp" is properly parsed as not ="some regexp". Previously it was incorrectly parsed as '~="some regexp"' phrase filter.

  • BUGFIX: properly sort results by _time field when limit pipe is applied. For example, _time:5m | sort by (_time) desc | limit 10 properly works now.

v0.9.1 #

Released at 2024-05-22

  • BUGFIX: web UI: fix loading web UI, which has been broken in v0.9.0.

v0.9.0 #

Released at 2024-05-22

  • FEATURE: allow using ~"some_regexp" regexp filter instead of re("some_regexp").

  • FEATURE: allow using ="some phrase" exact filter instead of exact("some phrase").

  • FEATURE: allow using ="some prefix"* exact prefix filter instead of exact("some prefix"*).

  • FEATURE: add ability to generate output fields according to the provided format string. See these docs.

  • FEATURE: add ability to extract fields with extract pipe only if the given condition is met. See these docs.

  • FEATURE: add ability to unpack JSON fields with unpack_json pipe only if the given condition is met. See these docs.

  • FEATURE: add ability to unpack logfmt fields with unpack_logfmt pipe only if the given condition is met. See these docs.

  • FEATURE: add row_min and row_max functions for stats pipe, which allow returning all the log fields for the log entry with the minimum / maximum value at the given field.

  • FEATURE: add /select/logsql/streams HTTP endpoint for returning streams from results of the given query. See these docs for details.

  • FEATURE: add /select/logsql/stream_field_names HTTP endpoint for returning stream field names from results of the given query. See these docs for details.

  • FEATURE: add /select/logsql/stream_field_values HTTP endpoint for returning stream field values for the given label from results of the given query. See these docs for details.

  • FEATURE: web UI: change time range limitation from _time in the expression to start and end query args.

  • BUGFIX: fix invalid memory address or nil pointer dereference panic when using extract, unpack_json or unpack_logfmt pipes. See this issue.

  • BUGFIX: web UI: fix an issue where logs with long _msg values might not display. See this issue.

  • BUGFIX: properly handle time range boundaries with millisecond precision. See this issue.

v0.8.0 #

Released at 2024-05-20

  • FEATURE: add ability to extract JSON fields from log fields. See these docs.

  • FEATURE: add ability to extract logfmt fields from log fields. See these docs.

  • FEATURE: add ability to extract arbitrary text from log fields into the output fields. See these docs.

  • FEATURE: add ability to put arbitrary queries inside in() filter.

  • FEATURE: add support for post-filtering of query results with filter pipe.

  • FEATURE: allow applying individual filters per each stats function. See these docs.

  • FEATURE: allow passing string values to min and max functions. Previously only numeric values could be passed to them.

  • FEATURE: speed up sort ... limit N pipe for typical cases.

  • FEATURE: allow using more convenient syntax for range filters if upper or lower bound isn’t needed. For example, it is possible to write response_size:>=10KiB instead of response_size:range[10KiB, inf), or temperature:<42 instead of temperature:range(-inf, 42).

  • FEATURE: add /select/logsql/hits HTTP endpoint for returning the number of matching logs per the given time bucket over the selected time range. See these docs for details.

  • FEATURE: add /select/logsql/field_names HTTP endpoint for returning field names from results of the given query. See these docs for details.

  • FEATURE: add /select/logsql/field_values HTTP endpoint for returning unique values for the given field obtained from results of the given query. See these docs for details.

  • BUGFIX: properly take into account offset at sort pipe when it already has limit. For example, _time:5m | sort by (foo) offset 20 limit 10.

v0.7.0 #

Released at 2024-05-15

  • FEATURE: add support for optional start and end query args to HTTP querying API, which can be used for limiting the time range for LogsQL query.
  • FEATURE: add ability to return the first N results from sort pipe. This is useful when N biggest or N smallest values must be returned from large amounts of logs.
  • FEATURE: add quantile and median stats functions.

v0.6.1 #

Released at 2024-05-14

v0.6.0 #

Released at 2024-05-12

  • FEATURE: return all the log fields by default in query results. Previously only _stream, _time and _msg fields were returned by default.

  • FEATURE: add support for returning only the requested log fields. See these docs.

  • FEATURE: add support for calculating various stats over log fields. Grouping by arbitrary set of log fields is supported. See these docs for details.

  • FEATURE: add support for sorting the returned results. See these docs.

  • FEATURE: add support for returning unique results. See these docs.

  • FEATURE: add support for limiting the number of returned results. See these docs.

  • FEATURE: add support for copying and renaming the selected log fields. See these and these docs.

  • FEATURE: allow using _ inside numbers. For example, score:range[1_000, 5_000_000] for range filter.

  • FEATURE: allow numbers in hexadecimal and binary form. For example, response_size:range[0xff, 0b10001101101] for range filter.

  • FEATURE: allow using duration and byte size suffixes in numeric values inside LogsQL queries. See these docs.

  • FEATURE: improve data ingestion performance by up to 50%.

  • FEATURE: optimize performance for LogsQL query, which contains multiple filters for words or phrases delimited with AND operator. For example, foo AND bar query must find log messages with foo and bar words at faster speed.

  • BUGFIX: prevent from possible corruption of short log fields during data ingestion.

  • BUGFIX: prevent from additional CPU usage for up to a few seconds after canceling the query.

  • BUGFIX: prevent from returning log entries with empty _stream field in the form "_stream":"" in search query results. See this issue.

v0.5.2 #

Released at 2024-04-11

  • BUGFIX: properly register new log streams under high data ingestion rate. The issue has been introduced in v0.5.0.

v0.5.1 #

Released at 2024-04-04

  • BUGFIX: properly apply time range filter for queries containing OR operators. See this issue.
  • BUGFIX: do not log debug lines DEBUG: start trimLines and DEBUG: end trimLines. This bug has been introduced in v0.5.0 in this commit.

v0.5.0 #

Released at 2024-03-01

  • FEATURE: support the ability to limit the number of returned log entries from HTTP querying API by passing limit query arg. Previously all the matching log entries were returned until closing the response stream. See this feature request. Thanks to @dmitryk-dk for the pull request.

  • BUGFIX: do not panic on incorrect regular expression in stream filter. Thanks to @XLONG96 for the bugfix.

  • BUGFIX: properly determine when the assisted merge is needed. Previously the logs for determining whether the assisted merge is needed was broken. This could lead to too big number of parts under high data ingestion rate. Thanks to @lujiajing1126 for the fix.

  • BUGFIX: properly stop execution of aborted query when the query doesn’t contain _stream filter. Previously such a query could continue consuming resources after being aborted by the client. Thanks to @z-anshun for the fix.

v0.4.2 #

Released at 2023-11-15

v0.4.1 #

Released at 2023-10-04

  • BUGFIX: fix the free space verification process in VictoriaLogs that was erroneously shifting to read-only mode, despite there being sufficient free space available. See this issue.

v0.4.0 #

Released at 2023-10-03

  • FEATURE: add -elasticsearch.version command-line flag, which can be used for specifying Elasticsearch version returned by VictoriaLogs to Filebeat at elasticsearch bulk API. This helps resolving this issue.

  • FEATURE: expose the following metrics at /metrics page:

    • vl_data_size_bytes{type="storage"} - on-disk size for data excluding log stream indexes.
    • vl_data_size_bytes{type="indexdb"} - on-disk size for log stream indexes.

  • FEATURE: add -insert.maxFieldsPerLine command-line flag, which can be used for limiting the number of fields per line in logs sent to VictoriaLogs via ingestion protocols. This helps to avoid issues like this.

  • FEATURE: expose vl_http_request_duration_seconds histogram at the /metrics page. Thanks to @crossoverJie for this pull request.

  • FEATURE: add support of -storage.minFreeDiskSpaceBytes command-line flag to allow switching to read-only mode when running out of disk space at -storageDataPath. See this issue.

  • BUGFIX: fix possible panic when no data is written to VictoriaLogs for a long time. See this issue. Thanks to @crossoverJie for filing and fixing the issue.

  • BUGFIX: add /insert/loki/ready endpoint, which is used by Promtail for healthchecks. This should remove unsupported path requested: /insert/loki/ready warning logs. See this comment.

  • BUGFIX: prevent from panic during background merge when the number of columns in the resulting block exceeds the maximum allowed number of columns per block. See this issue.

v0.3.0 #

Released at 2023-07-20

  • FEATURE: add support for data ingestion via Promtail (aka default log shipper for Grafana Loki). See these and these docs.

v0.2.0 #

Released at 2023-07-17

  • FEATURE: support short form of _time filters over the last X minutes/hours/days/etc. For example, _time:5m is a short form for _time:(now-5m, now], which matches logs with timestamps for the last 5 minutes. See these docs for details.
  • FEATURE: add ability to specify offset for the selected time range. For example, _time:5m offset 1h is equivalent to _time:(now-5m-1h, now-1h]. See these docs for details.
  • FEATURE: LogsQL: replace exact_prefix("...") with exact("..."*). This makes it consistent with i() filter, which can accept phrases and prefixes, e.g. i("phrase") and i("phrase"*). See these docs.

v0.1.0 #

Released at 2023-06-21

Initial release

Content

Back to Page