Data ingestion #

VictoriaTraces can accept trace spans via the OpenTelemetry protocol (OTLP) .

HTTP APIs #

Opentelemetry API #

VictoriaTraces provides the following API for OpenTelemetry data ingestion:

• /insert/opentelemetry/v1/traces

See more details in this docs .

HTTP parameters #

VictoriaTraces accepts optional HTTP parameters at data ingestion HTTP API via HTTP query string parameters , or via HTTP headers .

HTTP query string parameters have priority over HTTP Headers.

HTTP Query string parameters #

All the HTTP-based data ingestion protocols support the following HTTP query string args:

• extra_fields - an optional comma-separated list of trace fields , which must be added to all the ingested traces. The format of every extra_fields entry is field_name=field_value. If the trace entry contains fields from the extra_fields, then they are overwritten by the values specified in extra_fields.

• debug - if this arg is set to 1, then the ingested traces aren’t stored in VictoriaTraces. Instead, the ingested data is traceged by VictoriaTraces, so it can be investigated later.

See also HTTP headers .

HTTP headers #

All the HTTP-based data ingestion protocols support the following HTTP Headers additionally to HTTP query args :

• AccountID - accountID of the tenant to ingest data to. See multitenancy docs for details.

• ProjectID- projectID of the tenant to ingest data to. See multitenancy docs for details.

• VL-Extra-Fields - an optional comma-separated list of trace fields , which must be added to all the ingested traces. The format of every extra_fields entry is field_name=field_value. If the trace entry contains fields from the extra_fields, then they are overwritten by the values specified in extra_fields.

• VL-Debug - if this parameter is set to 1, then the ingested traces aren’t stored in VictoriaTraces. Instead, the ingested data is traceged by VictoriaTraces, so it can be investigated later.

See also HTTP Query string parameters .

Troubleshooting #

The following command can be used for verifying whether the data is successfully ingested into VictoriaTraces:

      curl http://<victoria-traces>:9428/select/logsql/query -d 'query=*' | head
    

This command selects all the data ingested into VictoriaTraces via HTTP query API using any value filter , while head cancels query execution after reading the first 10 trace spans. See these docs for more details on how head integrates with VictoriaTraces.

The response by default contains all the trace span fields . See how to query specific fields .

VictoriaTraces provides the following command-line flags, which can help debugging data ingestion issues:

• -logNewStreams - if this flag is passed to VictoriaTraces, then it traces all the newly registered streams . This may help debugging high cardinality issues .
• -logIngestedRows - if this flag is passed to VictoriaTraces, then it traces all the ingested trace span entries . See also debug parameter .

VictoriaTraces exposes various metrics , which may help debugging data ingestion issues:

• vl_rows_ingested_total - the number of ingested trace span entries since the last VictoriaTraces restart. If this number increases over time, then trace spans are successfully ingested into VictoriaTraces. The ingested trace spans can be inspected in the following ways:
  • By passing debug=1 parameter to every request to data ingestion APIs . The ingested spans aren’t stored in VictoriaTraces in this case. Instead, they are logged, so they can be investigated later. The vl_rows_dropped_total metric is incremented for each logged row.
  • By passing -logIngestedRows command-line flag to VictoriaTraces. In this case it traces all the ingested data, so it can be investigated later.
• vl_streams_created_total - the number of created trace streams since the last VictoriaTraces restart. If this metric grows rapidly during extended periods of time, then this may lead to high cardinality issues . The newly created trace streams can be inspected in traces by passing -logNewStreams command-line flag to VictoriaTraces.