Troubleshooting Grafana's Query Builder With VictoriaLogs

Dec 2, 2025 by Alex Johnson 58 views

Are you experiencing issues with Grafana's query builder not listing fields when using the VictoriaLogs data source? You're not alone! This article dives into a peculiar bug encountered while using the beta feature of the VictoriaLogs data source in Grafana. We'll explore the problem, how to reproduce it, and potential solutions. Let's get started!

Understanding the Grafana Query Builder Issue with VictoriaLogs

The core issue revolves around Grafana's query builder failing to display available fields for filtering after log fields have been cleared using the Opentelemetry Collector. This can be frustrating, especially when you need to drill down into your logs based on specific fields.

The user who reported this issue observed that after streamlining their log fields via Opentelemetry Collector, the query builder in Grafana stopped listing the available fields. Despite having numerous fields present in the logs, the dropdown in the query builder remained empty. Here’s a snippet of the fields the user had:

{
  "hits": "119224",
  "name": "stream"
}
{
  "hits": "119224",
  "name": "level"
}
{
  "hits": "119224",
  "name": "_stream_id"
}
...

The presence of these fields clearly indicates that data is available, but Grafana's query builder isn't picking them up. This discrepancy can significantly hinder the log exploration and analysis process. The user emphasized that while manually constructing queries works, the absence of the dropdown makes the process less intuitive and more prone to errors. In the subsequent sections, we'll delve deeper into how to reproduce this issue and explore potential causes and solutions.

Reproducing the Bug: A Step-by-Step Guide

To effectively troubleshoot this issue, it’s crucial to understand how to reproduce it. Here’s a detailed guide based on the user's report:

Set up Opentelemetry Collector: Configure the Opentelemetry Collector to report metrics to VictoriaLogs. The user provided a specific configuration for this, which we'll dissect in the next step.

Opentelemetry Collector Configuration: Use the following YAML configuration for your Opentelemetry Collector:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: open-telemetry
spec:
  mode: daemonset
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.140.1
  resources: {}
  observability:
    metrics:
      enableMetrics: true
  env:
    - name: POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

  targetAllocator:
    enabled: true
    image: ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.138.0
    resources: {}
    allocationStrategy: per-node
    prometheusCR:
      enabled: true
      scrapeInterval: 30s
      serviceMonitorSelector: {}

  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods
  volumeMounts:
    - name: varlogpods
      mountPath: /var/log/pods
  config:
    extensions:
      health_check:
        endpoint: ${env:POD_IP}:13133

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:43177
          http:
            endpoint: 0.0.0.0:43188
      filelog:
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        retry_on_failure:
          enabled: true
        start_at: beginning
        operators:
        - id: parser-containerd
          type: regex_parser
          regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
          timestamp:
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'
            parse_from: attributes.time

        - id: parser-pod-info
          parse_from: attributes["log.file.path"]
          regex: ^.*\[(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/([^_]+)\/([0-9]+)\.log$
          type: regex_parser

        - type: recombine
          is_last_entry: attributes.logtag == 'F'
          combine_field: attributes.log
          combine_with: ""
          max_batch_size: 1000
          max_log_size: 1048576
          output: handle_empty_log
          source_identifier: attributes["log.file.path"]
        - field: attributes.log
          id: handle_empty_log
          if: attributes.log == nil
          type: add
          value: ""

        - type: json_parser
          parse_from: attributes.log
          if: attributes.log matches "^\\{"

        - type: add
          field: attributes.instance
          value: ${env:K8S_NODE_NAME}

        - id: export
          type: noop

    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_max_size: 2048
        send_batch_size: 1024
        timeout: 1s
      transform/logs:
        error_mode: ignore
        log_statements:
          - statements:
            - set(log.attributes["namespace"], resource.attributes["namespace"])
            - keep_matching_keys(log.attributes, "^(_.*|@.*|filename|log|service|job|agent|k8s\\.|container_name|instance|level|msg|message|namespace|pod_name|severity|severity_text|stream)")
            - delete_matching_keys(log.attributes, "^(jobName|logger|loggerName|loggerClassName){{content}}quot;)
          - conditions: IsMap(log.body)
            statements:
              - keep_matching_keys(log.body, "^(level|msg|message|namespace|severity|severity_text){{content}}quot;)

    exporters:
      debug: {}
      otlphttp/victoriametrics:
        compression: gzip
        encoding: proto
        logs_endpoint: http://vmlogs-insert.victoriametrics:9481/insert/opentelemetry/v1/logs
        tls:
          insecure: true

    service:
      telemetry:
        logs:
          encoding: json
          level: info

      extensions:
        - health_check

      pipelines:
        logs:
          receivers: [filelog, otlp]
          processors:
            - memory_limiter
            - transform/logs
            - batch
          exporters: [otlphttp/victoriametrics]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug]
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug]

This configuration is crucial as it dictates how logs are collected, processed, and exported to VictoriaLogs. The transform/logs processor, in particular, plays a significant role in sanitizing the logs, which seems to be a trigger for the bug.

Deploy VictoriaLogs: Set up VictoriaLogs using the provided kustomization.yaml, vmauth-values.yaml, and vmlogs-values.yaml files. These configurations define the deployment specifications for VictoriaLogs, including authentication, resource allocation, and storage.
Clear Log Fields: After the metrics are reported to VictoriaLogs, clear most of the log fields using the Opentelemetry Collector. This is a key step in triggering the bug.
Access Grafana: Open Grafana and navigate to the query builder for the VictoriaLogs data source.
Observe the Bug: Check if the query builder fails to list the available fields to filter. If the bug is successfully reproduced, the dropdown should be empty despite the presence of log data.

By following these steps, you should be able to reproduce the bug and verify that the query builder isn't listing fields as expected. Once the bug is consistently reproducible, it becomes easier to explore potential causes and solutions.

VictoriaLogs and Grafana Versions

Understanding the specific versions of VictoriaLogs, Grafana, and the VictoriaLogs datasource is crucial for troubleshooting. Here's the version information from the user's report:

Grafana: v12.3.0 (20051fb1fc)
VictoriaLogs Datasource: v0.22.3
VictoriaLogs: v1.38.0

These versions provide a specific context for the issue. It's possible that the bug is specific to these versions or a combination thereof. When reporting issues or seeking help, always include version information to ensure that others can accurately understand and address the problem.

Potential Causes and Troubleshooting Steps

Several factors could contribute to Grafana's query builder failing to list fields in the VictoriaLogs data source. Here are some potential causes and troubleshooting steps:

Data Sanitization: The Opentelemetry Collector configuration includes a transform/logs processor that sanitizes logs by keeping and deleting specific keys. This process might inadvertently remove metadata required by Grafana to list fields. Review the transform/logs configuration in the Opentelemetry Collector YAML file. Ensure that essential fields are not being inadvertently dropped during the sanitization process. The keep_matching_keys and delete_matching_keys statements should be carefully examined to confirm that they are not removing fields needed by Grafana.
Field Discovery: Grafana's query builder relies on field discovery to populate the dropdown list. If the data source is not correctly configured to discover fields, the list will remain empty. Check the VictoriaLogs data source configuration in Grafana. Ensure that the data source is correctly configured to discover fields. This might involve checking connection settings, query parameters, and any specific settings related to field discovery.
Caching Issues: Grafana might be caching an outdated schema or field list. Clearing the cache might resolve the issue. Try clearing Grafana's cache or restarting the Grafana server. This can help ensure that Grafana is fetching the latest data and schema information from VictoriaLogs.
Data Volume: If the volume of log data is very high, it might take longer for Grafana to fetch and process the fields. Increase the query timeout in the Grafana data source settings. This might give Grafana more time to fetch the field list, especially if the data volume is high.
Backend Issues: There might be an issue with the VictoriaLogs backend that prevents Grafana from fetching the field list. Check the VictoriaLogs logs for any errors or warnings. This can provide insights into whether VictoriaLogs is encountering any issues while processing or serving the data.
Beta Feature Bugs: Since the VictoriaLogs data source is in beta, there might be undiscovered bugs. Report the issue to the VictoriaLogs and Grafana communities. Providing detailed information, including reproduction steps and version numbers, can help developers identify and fix the bug.

By systematically investigating these potential causes, you can narrow down the source of the problem and implement the appropriate solution. In the next section, we'll discuss some additional steps and considerations for resolving this issue.

Additional Steps and Considerations

When troubleshooting issues with Grafana and VictoriaLogs, consider these additional steps and factors:

Check VictoriaLogs Logs: Review the logs from VictoriaLogs components (vlinsert, vlselect, vlstorage) to identify any errors or warnings. These logs can provide valuable insights into potential issues on the backend. Look for any error messages, connection problems, or performance bottlenecks that might be affecting field discovery.
Grafana Data Source Settings: Double-check the VictoriaLogs data source settings in Grafana. Ensure that the URL, authentication details, and other settings are correctly configured. Incorrect settings can prevent Grafana from properly connecting to and querying VictoriaLogs.
Network Connectivity: Verify that there are no network connectivity issues between Grafana and VictoriaLogs. Use tools like ping or traceroute to check network paths. Firewalls or network policies might be blocking communication between Grafana and VictoriaLogs.
Resource Limits: Ensure that VictoriaLogs has sufficient resources (CPU, memory, disk I/O) to handle the queries from Grafana. Resource constraints can lead to performance issues and prevent field discovery. Monitor the resource usage of VictoriaLogs components and adjust limits as necessary.
Query Performance: If field discovery is slow, it might be due to inefficient queries. Optimize your queries in VictoriaLogs to improve performance. Use indexes and other optimization techniques to speed up data retrieval.
Community Support: Engage with the VictoriaLogs and Grafana communities for assistance. Share your issue on forums, mailing lists, or chat channels. Other users might have encountered the same problem and can offer solutions or workarounds.
Simplified Configuration: Try simplifying your Opentelemetry Collector configuration to isolate the issue. Remove unnecessary processors or exporters to see if the problem persists. This can help determine if a specific component is causing the bug.

By considering these additional steps and factors, you can take a comprehensive approach to troubleshooting the issue and increase your chances of finding a solution.

Conclusion

The issue of Grafana's query builder failing to list fields when using the VictoriaLogs data source can be a significant hurdle in log analysis. By understanding the bug, how to reproduce it, potential causes, and troubleshooting steps, you can effectively address the problem. Remember to check your configurations, monitor logs, engage with the community, and systematically investigate potential causes. While the VictoriaLogs data source is in beta, reporting issues and sharing your experiences can help improve the product for everyone.

For further reading on Grafana and VictoriaLogs, consider exploring the official documentation and community resources. You might find helpful information on the VictoriaMetrics Documentation.