VictoriaTraces is currently under active development and not ready for production use. It is built on top of VictoriaLogs and therefore shares some flags and APIs. These will be fully separated once VictoriaTraces reaches a stable release. Until then, features may change or break without notice.
VictoriaTraces can accept trace spans via the OpenTelemetry protocol (OTLP) .
HTTP APIs #
Opentelemetry API #
VictoriaTraces provides the following API for OpenTelemetry data ingestion:
/insert/opentelemetry/v1/traces
See more details in this docs .
HTTP parameters #
VictoriaTraces accepts optional HTTP parameters at data ingestion HTTP API via HTTP query string parameters , or via HTTP headers .
HTTP query string parameters have priority over HTTP Headers.
HTTP Query string parameters #
All the HTTP-based data ingestion protocols support the following HTTP query string args:
extra_fields
- an optional comma-separated list of trace fields , which must be added to all the ingested traces. The format of everyextra_fields
entry isfield_name=field_value
. If the trace entry contains fields from theextra_fields
, then they are overwritten by the values specified inextra_fields
.debug
- if this arg is set to1
, then the ingested traces aren’t stored in VictoriaTraces. Instead, the ingested data is traceged by VictoriaTraces, so it can be investigated later.
See also HTTP headers .
HTTP headers #
All the HTTP-based data ingestion protocols support the following HTTP Headers additionally to HTTP query args :
AccountID
- accountID of the tenant to ingest data to. See multitenancy docs for details.ProjectID
- projectID of the tenant to ingest data to. See multitenancy docs for details.VL-Extra-Fields
- an optional comma-separated list of trace fields , which must be added to all the ingested traces. The format of everyextra_fields
entry isfield_name=field_value
. If the trace entry contains fields from theextra_fields
, then they are overwritten by the values specified inextra_fields
.VL-Debug
- if this parameter is set to1
, then the ingested traces aren’t stored in VictoriaTraces. Instead, the ingested data is traceged by VictoriaTraces, so it can be investigated later.
See also HTTP Query string parameters .
Troubleshooting #
The following command can be used for verifying whether the data is successfully ingested into VictoriaTraces:
curl http://<victoria-traces>:10428/select/logsql/query -d 'query=*' | head
This command selects all the data ingested into VictoriaTraces via HTTP query API
using any value filter
,
while head
cancels query execution after reading the first 10 trace spans. See these docs
for more details on how head
integrates with VictoriaTraces.
The response by default contains all the trace span fields . See how to query specific fields .
VictoriaTraces provides the following command-line flags, which can help debugging data ingestion issues:
-logNewStreams
- if this flag is passed to VictoriaTraces, then it traces all the newly registered streams . This may help debugging high cardinality issues .-logIngestedRows
- if this flag is passed to VictoriaTraces, then it traces all the ingested trace span entries . See alsodebug
parameter .
VictoriaTraces exposes various metrics, which may help debugging data ingestion issues:
vt_rows_ingested_total
- the number of ingested trace span entries since the last VictoriaTraces restart. If this number increases over time, then trace spans are successfully ingested into VictoriaTraces. The ingested trace spans can be inspected in the following ways:- By passing
debug=1
parameter to every request to data ingestion APIs . The ingested spans aren’t stored in VictoriaTraces in this case. Instead, they are logged, so they can be investigated later. Thevt_rows_dropped_total
metric is incremented for each logged row. - By passing
-logIngestedRows
command-line flag to VictoriaTraces. In this case it traces all the ingested data, so it can be investigated later.
- By passing
vt_streams_created_total
- the number of created trace streams since the last VictoriaTraces restart. If this metric grows rapidly during extended periods of time, then this may lead to high cardinality issues . The newly created trace streams can be inspected in traces by passing-logNewStreams
command-line flag to VictoriaTraces.