我有一個 AKS 集群，正在其中運行基於 Python-Django 的測試 Web 應用程式。我還配置了 Grafana 和 Prometheus。我需要使用 OpenTelemetry 從測試應用程式獲取指標資料並在 Grafana 中對其進行視覺化。
我使用 OTEL CRD 以 Kubernetes 運算子的形式部署了 OpenTelemetry Collector（請參閱：https://opentelemetry.io/docs/kubernetes/operator/automatic/）。
我已經部署了 OpenTelemetry CRD 並部署了收集器。收藏家很善良OpenTelemetryCollector。我已將匯出器配置為PromethusOTEL 收集器為 Prometheus 提供服務，並且全部已連接。我可以在 Grafana 中查看基於 OTEL 收集器的面板。
我的目標是使用 OTEL 收集器在 Grafana 中獲取基於應用程式的指標，例如pod_cpu_usage、等。pod_memory_usage
以下是我的 OTEL Collector YAML 設定：
otel-collector.yml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: otel
labels:
opentelemetry.io/name: otelcol
app.kubernetes.io/instance: opentelemetry
app.kubernetes.io/name: otelcol
app.kubernetes.io/version: "0.82.0"
annotations:
otel_collector: "true"
prometheus.io/port: "9464"
prometheus.io/scrape: "true"
spec:
config: |
receivers:
prometheus:
config:
scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: integrations/kubernetes/cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc.cluster.local:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: integrations/kubernetes/kubelet
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc.cluster.local:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$${1}/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- job_name: integrations/kubernetes/kube-state-metrics
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: kube-state-metrics
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- job_name: integrations/node_exporter
kubernetes_sd_configs:
- namespaces:
names:
- default
role: pod
relabel_configs:
- action: keep
regex: prometheus-node-exporter.*
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: instance
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
otlp:
protocols:
grpc:
endpoint: '0.0.0.0:4317'
http:
cors:
allowed_origins:
- http://*
- https://*
endpoint: '0.0.0.0:4318'
processors:
batch: {}
filter/ottl:
error_mode: ignore
metrics:
metric:
- name == "rpc.server.duration"
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
resource:
attributes:
- action: insert
from_attribute: k8s.pod.uid
key: service.instance.id
exporters:
logging: {}
prometheus:
enable_open_metrics: true
endpoint: '0.0.0.0:9464'
resource_to_telemetry_conversion:
enabled: true
extensions:
health_check: {}
memory_ballast:
size_in_percentage: 40
service:
pipelines:
metrics:
receivers: [prometheus, otlp]
processors: [batch, filter/ottl, memory_limiter, resource]
exporters: [prometheus, logging]
以下是目前 Instrumentation 設定 YAML（參考：https://github.com/open-telemetry/opentelemetry-operator#deployment-modes） 我有：
otel-instrumentation.yml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: python-instrumentation
namespace: trojanwall
spec:
exporter:
endpoint: http://10.244.0.7:4318
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "1"
python:
env:
- name: OTEL_LOGS_EXPORTER
value: otlp_proto_http
- name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
value: 'true'
我需要在不修改任何應用程式原始碼的情況下為 Python-Django 應用程式實現自動檢測。
之後，我將以下註解新增至我的 Python-Django 測試應用程式部署 YAML 檔案：
annotations:
instrumentation.opentelemetry.io/inject-python: "true"
但是，Grafana 中沒有顯示我的應用程式特定指標。
如何使用 OpenTelemetry Collector 和自動偵測正確執行此操作？
有人可以提供有關如何正確執行此操作的任何見解嗎？