Still viable way to scrape Volume metrics? Https issue on K8s v1.13.3

[MisterTimn]

I stumbled on this project after figuring out how I could get PV metrics, apparently they can't be scraped from the /metrics endpoint of the kubelet anymore.

In some issue I found this project and deployed it on our cluster, however, kubelet endpoints are on https so there's need for some tlsconfig when you want to scrape these endpoints.
So unfortunately right now the /metrics endpoint of the kubelet-exporter gives me a blank page now, because I think it cant reach the kubelet /stats/summary path.

Question is, are there any plans to add https to this exporter? I have no experience in Go otherwise I would fork this repo and try myself.
Or is there another way you know of now to get those Volume usage metrics?

[cofyc]

hi, originally this export is written to work around a metrics bug in kubelet. Now the bug is fixed, you should scrape /metrics path of kubelet directly.

[lianghuiyuan]

It seems K8S 1.14 not fixed the bug?
scrape /metrics path of kubelet directly can't get kubelet_volume* metrics
and I deploy 'kubelet-exporter' can not scrape the kubelet_volume* metrics either

[lianghuiyuan]

$ kubectl exec -it centos-tool-auto-84c77c49bd-knhw5 -c centos -n lateinos  -- curl -X GET http://kubelet.kube-system.svc:10255/metrics
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="3600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="7200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="7.776e+06"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1.5552e+07"} 22
apiserver_client_certificate_expiration_seconds_bucket{le="3.1104e+07"} 22
apiserver_client_certificate_expiration_seconds_bucket{le="+Inf"} 22
apiserver_client_certificate_expiration_seconds_sum 3.2237449300073373e+08
apiserver_client_certificate_expiration_seconds_count 22
# HELP apiserver_storage_data_key_generation_duration_seconds Latencies in seconds of data encryption key(DEK) generation operations.
# TYPE apiserver_storage_data_key_generation_duration_seconds histogram
apiserver_storage_data_key_generation_duration_seconds_bucket{le="5e-06"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="1e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="2e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="4e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="8e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00016"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00032"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00064"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00128"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00256"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00512"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.01024"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.02048"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.04096"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="+Inf"} 0
apiserver_storage_data_key_generation_duration_seconds_sum 0
apiserver_storage_data_key_generation_duration_seconds_count 0
# HELP apiserver_storage_data_key_generation_failures_total Total number of failed data encryption key(DEK) generation operations.
# TYPE apiserver_storage_data_key_generation_failures_total counter
apiserver_storage_data_key_generation_failures_total 0
# HELP apiserver_storage_data_key_generation_latencies_microseconds (Deprecated) Latencies in microseconds of data encryption key(DEK) generation operations.
# TYPE apiserver_storage_data_key_generation_latencies_microseconds histogram
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="5"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="10"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="20"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="40"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="80"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="160"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="320"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="640"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="1280"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="2560"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="5120"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="10240"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="20480"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="40960"} 0
apiserver_storage_data_key_generation_latencies_microseconds_bucket{le="+Inf"} 0
apiserver_storage_data_key_generation_latencies_microseconds_sum 0
apiserver_storage_data_key_generation_latencies_microseconds_count 0
# HELP apiserver_storage_envelope_transformation_cache_misses_total Total number of cache misses while accessing key decryption key(KEK).
# TYPE apiserver_storage_envelope_transformation_cache_misses_total counter
apiserver_storage_envelope_transformation_cache_misses_total 0
# HELP get_token_count Counter of total Token() requests to the alternate token source
# TYPE get_token_count counter
get_token_count 0
# HELP get_token_fail_count Counter of failed Token() requests to the alternate token source
# TYPE get_token_fail_count counter
get_token_fail_count 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.3983e-05
go_gc_duration_seconds{quantile="0.25"} 2.0286e-05
go_gc_duration_seconds{quantile="0.5"} 2.535e-05
go_gc_duration_seconds{quantile="0.75"} 4.1696e-05
go_gc_duration_seconds{quantile="1"} 0.002793496
go_gc_duration_seconds_sum 1.849114298
go_gc_duration_seconds_count 33187
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 441
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12.10"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 8.3257056e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.667467954496e+12
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 5.551915e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.4264748678e+10
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0013608462858577493
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 7.620608e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 8.3257056e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 9.3208576e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.03661568e+08
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 620107
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.169728e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.96870144e+08
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5762182916899424e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.4265368785e+10
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 6944
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 1.516464e+06
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 1.736704e+06
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.14138304e+08
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.102285e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 4.456448e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 4.456448e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.17354488e+08
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 25
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 19417.107
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 22628.226
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 24722.927
http_request_duration_microseconds_sum{handler="prometheus"} 2.6774948899699894e+08
http_request_duration_microseconds_count{handler="prometheus"} 15596
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 1116
http_request_size_bytes{handler="prometheus",quantile="0.9"} 1116
http_request_size_bytes{handler="prometheus",quantile="0.99"} 1116
http_request_size_bytes_sum{handler="prometheus"} 1.7405136e+07
http_request_size_bytes_count{handler="prometheus"} 15596
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 15596
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 12279
http_response_size_bytes{handler="prometheus",quantile="0.9"} 12294
http_response_size_bytes{handler="prometheus",quantile="0.99"} 12298
http_response_size_bytes_sum{handler="prometheus"} 1.88023591e+08
http_response_size_bytes_count{handler="prometheus"} 15596
# HELP kubelet_certificate_manager_client_expiration_seconds Gauge of the lifetime of a certificate. The value is the date the certificate will expire in seconds since January 1, 1970 UTC.
# TYPE kubelet_certificate_manager_client_expiration_seconds gauge
kubelet_certificate_manager_client_expiration_seconds 1.59064422e+09
# HELP kubelet_cgroup_manager_duration_seconds Duration in seconds for cgroup manager operations. Broken down by method.
# TYPE kubelet_cgroup_manager_duration_seconds histogram
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.005"} 0
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.01"} 0
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.025"} 27
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.05"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.1"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.25"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.5"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="1"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="2.5"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="5"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="10"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="+Inf"} 33
kubelet_cgroup_manager_duration_seconds_sum{operation_type="create"} 0.6138125509999999
kubelet_cgroup_manager_duration_seconds_count{operation_type="create"} 33
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="0.005"} 0
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="0.01"} 19
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="0.025"} 28
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="0.05"} 30
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="0.1"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="0.25"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="0.5"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="1"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="2.5"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="5"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="10"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="destroy",le="+Inf"} 32
kubelet_cgroup_manager_duration_seconds_sum{operation_type="destroy"} 0.4640743430000001
kubelet_cgroup_manager_duration_seconds_count{operation_type="destroy"} 32
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.005"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.01"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.025"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.05"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.1"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.25"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.5"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="1"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="2.5"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="5"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="10"} 8126
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="+Inf"} 8126
kubelet_cgroup_manager_duration_seconds_sum{operation_type="update"} 0.7082004070000009
kubelet_cgroup_manager_duration_seconds_count{operation_type="update"} 8126
# HELP kubelet_cgroup_manager_latency_microseconds (Deprecated) Latency in microseconds for cgroup manager operations. Broken down by method.
# TYPE kubelet_cgroup_manager_latency_microseconds summary
kubelet_cgroup_manager_latency_microseconds{operation_type="create",quantile="0.5"} NaN
kubelet_cgroup_manager_latency_microseconds{operation_type="create",quantile="0.9"} NaN
kubelet_cgroup_manager_latency_microseconds{operation_type="create",quantile="0.99"} NaN
kubelet_cgroup_manager_latency_microseconds_sum{operation_type="create"} 613876
kubelet_cgroup_manager_latency_microseconds_count{operation_type="create"} 33
kubelet_cgroup_manager_latency_microseconds{operation_type="destroy",quantile="0.5"} NaN
kubelet_cgroup_manager_latency_microseconds{operation_type="destroy",quantile="0.9"} NaN
kubelet_cgroup_manager_latency_microseconds{operation_type="destroy",quantile="0.99"} NaN
kubelet_cgroup_manager_latency_microseconds_sum{operation_type="destroy"} 464176
kubelet_cgroup_manager_latency_microseconds_count{operation_type="destroy"} 32
kubelet_cgroup_manager_latency_microseconds{operation_type="update",quantile="0.5"} 69
kubelet_cgroup_manager_latency_microseconds{operation_type="update",quantile="0.9"} 112
kubelet_cgroup_manager_latency_microseconds{operation_type="update",quantile="0.99"} 126
kubelet_cgroup_manager_latency_microseconds_sum{operation_type="update"} 718652
kubelet_cgroup_manager_latency_microseconds_count{operation_type="update"} 8126
# HELP kubelet_container_log_filesystem_used_bytes Bytes used by the container's logs on the filesystem.
# TYPE kubelet_container_log_filesystem_used_bytes gauge
kubelet_container_log_filesystem_used_bytes{container="acs-flexvolume",namespace="kube-system",pod="flexvolume-xzpln"} 28672
kubelet_container_log_filesystem_used_bytes{container="alertmanager",namespace="monitoring",pod="alertmanager-main-1"} 53248
kubelet_container_log_filesystem_used_bytes{container="citadel",namespace="istio-system",pod="istio-citadel-554fff9959-429qd"} 143360
kubelet_container_log_filesystem_used_bytes{container="config-reloader",namespace="monitoring",pod="alertmanager-main-1"} 24576
kubelet_container_log_filesystem_used_bytes{container="directoryservice",namespace="h3yun-system",pod="directoryservice-auto-7475c75866-cpn2m"} 462848
kubelet_container_log_filesystem_used_bytes{container="galley",namespace="istio-system",pod="istio-galley-79995cff9b-qmdj8"} 1.298432e+06
kubelet_container_log_filesystem_used_bytes{container="istio-proxy",namespace="h3yun-system",pod="directoryservice-auto-7475c75866-cpn2m"} 794624
kubelet_container_log_filesystem_used_bytes{container="istio-proxy",namespace="istio-system",pod="istio-policy-c5b5d66d7-ws9mm"} 69632
kubelet_container_log_filesystem_used_bytes{container="istio-proxy",namespace="lateinos",pod="lateinos-monitor-rules-auto-74556b7984-whfdr"} 2.00835072e+08
kubelet_container_log_filesystem_used_bytes{container="istio-proxy",namespace="lateinos",pod="qconf-agent-ds-sf96p"} 212992
kubelet_container_log_filesystem_used_bytes{container="kube-proxy-worker",namespace="kube-system",pod="kube-proxy-worker-b75jb"} 876544
kubelet_container_log_filesystem_used_bytes{container="kube-rbac-proxy",namespace="monitoring",pod="node-exporter-8bm2m"} 28672
kubelet_container_log_filesystem_used_bytes{container="kubelet-exporter",namespace="monitoring",pod="kubelet-exporter-vkhhf"} 28672
kubelet_container_log_filesystem_used_bytes{container="lateinos-monitor-rules",namespace="lateinos",pod="lateinos-monitor-rules-auto-74556b7984-whfdr"} 1.0692608e+08
kubelet_container_log_filesystem_used_bytes{container="lateinos-nodejs-svcdemo",namespace="lateinos",pod="lateinos-nodejs-svcdemo-auto-c775d8b44-f4bvr"} 77824
kubelet_container_log_filesystem_used_bytes{container="logtail",namespace="kube-system",pod="logtail-ds-ppq6w"} 61440
kubelet_container_log_filesystem_used_bytes{container="mixer",namespace="istio-system",pod="istio-policy-c5b5d66d7-ws9mm"} 4.665344e+06
kubelet_container_log_filesystem_used_bytes{container="nginx-gateway-report-exporter",namespace="monitoring",pod="nginx-gateway-report-exporter-deployment-6f4f4bc57b-7mrns"} 491520
kubelet_container_log_filesystem_used_bytes{container="node-exporter",namespace="monitoring",pod="node-exporter-8bm2m"} 32768
kubelet_container_log_filesystem_used_bytes{container="policy",namespace="kube-system",pod="terway-dl95r"} 3.987456e+07
kubelet_container_log_filesystem_used_bytes{container="qconf-agent",namespace="lateinos",pod="qconf-agent-ds-sf96p"} 36864
kubelet_container_log_filesystem_used_bytes{container="reporter-integrate-webapi",namespace="reporter",pod="reporter-integrate-webapi-deploy-8464bf4fcd-s5tr7"} 2.3252992e+07
kubelet_container_log_filesystem_used_bytes{container="sidecar-injector-webhook",namespace="istio-system",pod="istio-sidecar-injector-7bc498695b-nxggg"} 3.52256e+06
kubelet_container_log_filesystem_used_bytes{container="terway",namespace="kube-system",pod="terway-dl95r"} 94208
# HELP kubelet_containers_per_pod_count The number of containers per pod.
# TYPE kubelet_containers_per_pod_count histogram
kubelet_containers_per_pod_count_bucket{le="0.005"} 0
kubelet_containers_per_pod_count_bucket{le="0.01"} 0
kubelet_containers_per_pod_count_bucket{le="0.025"} 0
kubelet_containers_per_pod_count_bucket{le="0.05"} 0
kubelet_containers_per_pod_count_bucket{le="0.1"} 0
kubelet_containers_per_pod_count_bucket{le="0.25"} 0
kubelet_containers_per_pod_count_bucket{le="0.5"} 0
kubelet_containers_per_pod_count_bucket{le="1"} 20
kubelet_containers_per_pod_count_bucket{le="2.5"} 49
kubelet_containers_per_pod_count_bucket{le="5"} 49
kubelet_containers_per_pod_count_bucket{le="10"} 49
kubelet_containers_per_pod_count_bucket{le="+Inf"} 49
kubelet_containers_per_pod_count_sum 78
kubelet_containers_per_pod_count_count 49
# HELP kubelet_device_plugin_registration_count (Deprecated) Cumulative number of device plugin registrations. Broken down by resource name.
# TYPE kubelet_device_plugin_registration_count counter
kubelet_device_plugin_registration_count{resource_name="aliyun/eni"} 1
# HELP kubelet_device_plugin_registration_total Cumulative number of device plugin registrations. Broken down by resource name.
# TYPE kubelet_device_plugin_registration_total counter
kubelet_device_plugin_registration_total{resource_name="aliyun/eni"} 1
# HELP kubelet_docker_operations (Deprecated) Cumulative number of Docker operations by operation type.
# TYPE kubelet_docker_operations counter
kubelet_docker_operations{operation_type="create_container"} 111
kubelet_docker_operations{operation_type="info"} 2
kubelet_docker_operations{operation_type="inspect_container"} 1272
kubelet_docker_operations{operation_type="inspect_image"} 1432
kubelet_docker_operations{operation_type="list_containers"} 869716
kubelet_docker_operations{operation_type="list_images"} 33350
kubelet_docker_operations{operation_type="pull_image"} 19
kubelet_docker_operations{operation_type="remove_container"} 114
kubelet_docker_operations{operation_type="start_container"} 111
kubelet_docker_operations{operation_type="stop_container"} 187
kubelet_docker_operations{operation_type="version"} 70559
# HELP kubelet_docker_operations_duration_seconds Latency in seconds of Docker operations. Broken down by operation type.
# TYPE kubelet_docker_operations_duration_seconds histogram
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="0.005"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="0.01"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="0.025"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="0.05"} 92
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="0.1"} 110
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="0.25"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="0.5"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="1"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="2.5"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="5"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="10"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="create_container",le="+Inf"} 111
kubelet_docker_operations_duration_seconds_sum{operation_type="create_container"} 4.6072620350000015
kubelet_docker_operations_duration_seconds_count{operation_type="create_container"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="0.005"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="0.01"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="0.025"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="0.05"} 1
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="0.1"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="0.25"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="0.5"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="1"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="2.5"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="5"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="10"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="info",le="+Inf"} 2
kubelet_docker_operations_duration_seconds_sum{operation_type="info"} 0.098034917
kubelet_docker_operations_duration_seconds_count{operation_type="info"} 2
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="0.005"} 1207
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="0.01"} 1221
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="0.025"} 1226
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="0.05"} 1231
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="0.1"} 1242
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="0.25"} 1256
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="0.5"} 1266
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="1"} 1270
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="2.5"} 1272
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="5"} 1272
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="10"} 1272
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_container",le="+Inf"} 1272
kubelet_docker_operations_duration_seconds_sum{operation_type="inspect_container"} 14.779075727000002
kubelet_docker_operations_duration_seconds_count{operation_type="inspect_container"} 1272
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="0.005"} 1416
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="0.01"} 1425
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="0.025"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="0.05"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="0.1"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="0.25"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="0.5"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="1"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="2.5"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="5"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="10"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="inspect_image",le="+Inf"} 1432
kubelet_docker_operations_duration_seconds_sum{operation_type="inspect_image"} 2.0499678730000004
kubelet_docker_operations_duration_seconds_count{operation_type="inspect_image"} 1432
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="0.005"} 307704
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="0.01"} 820370
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="0.025"} 868471
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="0.05"} 869699
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="0.1"} 869713
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="0.25"} 869713
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="0.5"} 869714
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="1"} 869715
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="2.5"} 869716
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="5"} 869716
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="10"} 869716
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_containers",le="+Inf"} 869716
kubelet_docker_operations_duration_seconds_sum{operation_type="list_containers"} 5406.22504373397
kubelet_docker_operations_duration_seconds_count{operation_type="list_containers"} 869716
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="0.005"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="0.01"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="0.025"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="0.05"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="0.1"} 30265
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="0.25"} 33350
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="0.5"} 33350
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="1"} 33350
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="2.5"} 33350
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="5"} 33350
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="10"} 33350
kubelet_docker_operations_duration_seconds_bucket{operation_type="list_images",le="+Inf"} 33350
kubelet_docker_operations_duration_seconds_sum{operation_type="list_images"} 2724.4517583369975
kubelet_docker_operations_duration_seconds_count{operation_type="list_images"} 33350
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="0.005"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="0.01"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="0.025"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="0.05"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="0.1"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="0.25"} 1
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="0.5"} 1
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="1"} 1
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="2.5"} 1
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="5"} 7
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="10"} 10
kubelet_docker_operations_duration_seconds_bucket{operation_type="pull_image",le="+Inf"} 19
kubelet_docker_operations_duration_seconds_sum{operation_type="pull_image"} 188.76506255500001
kubelet_docker_operations_duration_seconds_count{operation_type="pull_image"} 19
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="0.005"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="0.01"} 24
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="0.025"} 97
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="0.05"} 110
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="0.1"} 113
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="0.25"} 114
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="0.5"} 114
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="1"} 114
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="2.5"} 114
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="5"} 114
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="10"} 114
kubelet_docker_operations_duration_seconds_bucket{operation_type="remove_container",le="+Inf"} 114
kubelet_docker_operations_duration_seconds_sum{operation_type="remove_container"} 1.9736502369999995
kubelet_docker_operations_duration_seconds_count{operation_type="remove_container"} 114
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="0.005"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="0.01"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="0.025"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="0.05"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="0.1"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="0.25"} 0
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="0.5"} 75
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="1"} 110
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="2.5"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="5"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="10"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="start_container",le="+Inf"} 111
kubelet_docker_operations_duration_seconds_sum{operation_type="start_container"} 52.65310921700001
kubelet_docker_operations_duration_seconds_count{operation_type="start_container"} 111
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="0.005"} 105
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="0.01"} 105
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="0.025"} 105
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="0.05"} 105
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="0.1"} 105
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="0.25"} 107
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="0.5"} 144
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="1"} 144
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="2.5"} 144
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="5"} 146
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="10"} 168
kubelet_docker_operations_duration_seconds_bucket{operation_type="stop_container",le="+Inf"} 187
kubelet_docker_operations_duration_seconds_sum{operation_type="stop_container"} 710.673009789
kubelet_docker_operations_duration_seconds_count{operation_type="stop_container"} 187
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="0.005"} 70432
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="0.01"} 70548
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="0.025"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="0.05"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="0.1"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="0.25"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="0.5"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="1"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="2.5"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="5"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="10"} 70559
kubelet_docker_operations_duration_seconds_bucket{operation_type="version",le="+Inf"} 70559
kubelet_docker_operations_duration_seconds_sum{operation_type="version"} 47.75353757200034
kubelet_docker_operations_duration_seconds_count{operation_type="version"} 70559
# HELP kubelet_docker_operations_errors (Deprecated) Cumulative number of Docker operation errors by operation type.
# TYPE kubelet_docker_operations_errors counter
kubelet_docker_operations_errors{operation_type="inspect_container"} 46
kubelet_docker_operations_errors{operation_type="inspect_image"} 10
# HELP kubelet_docker_operations_errors_total Cumulative number of Docker operation errors by operation type.
# TYPE kubelet_docker_operations_errors_total counter
kubelet_docker_operations_errors_total{operation_type="inspect_container"} 46
kubelet_docker_operations_errors_total{operation_type="inspect_image"} 10
# HELP kubelet_docker_operations_latency_microseconds (Deprecated) Latency in microseconds of Docker operations. Broken down by operation type.
# TYPE kubelet_docker_operations_latency_microseconds summary
kubelet_docker_operations_latency_microseconds{operation_type="create_container",quantile="0.5"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="create_container",quantile="0.9"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="create_container",quantile="0.99"} NaN
kubelet_docker_operations_latency_microseconds_sum{operation_type="create_container"} 4.607601e+06
kubelet_docker_operations_latency_microseconds_count{operation_type="create_container"} 111
kubelet_docker_operations_latency_microseconds{operation_type="info",quantile="0.5"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="info",quantile="0.9"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="info",quantile="0.99"} NaN
kubelet_docker_operations_latency_microseconds_sum{operation_type="info"} 98067
kubelet_docker_operations_latency_microseconds_count{operation_type="info"} 2
kubelet_docker_operations_latency_microseconds{operation_type="inspect_container",quantile="0.5"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="inspect_container",quantile="0.9"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="inspect_container",quantile="0.99"} NaN
kubelet_docker_operations_latency_microseconds_sum{operation_type="inspect_container"} 1.4780983e+07
kubelet_docker_operations_latency_microseconds_count{operation_type="inspect_container"} 1272
kubelet_docker_operations_latency_microseconds{operation_type="inspect_image",quantile="0.5"} 1339
kubelet_docker_operations_latency_microseconds{operation_type="inspect_image",quantile="0.9"} 1339
kubelet_docker_operations_latency_microseconds{operation_type="inspect_image",quantile="0.99"} 1339
kubelet_docker_operations_latency_microseconds_sum{operation_type="inspect_image"} 2.052819e+06
kubelet_docker_operations_latency_microseconds_count{operation_type="inspect_image"} 1432
kubelet_docker_operations_latency_microseconds{operation_type="list_containers",quantile="0.5"} 6485
kubelet_docker_operations_latency_microseconds{operation_type="list_containers",quantile="0.9"} 9215
kubelet_docker_operations_latency_microseconds{operation_type="list_containers",quantile="0.99"} 16935
kubelet_docker_operations_latency_microseconds_sum{operation_type="list_containers"} 5.408210327e+09
kubelet_docker_operations_latency_microseconds_count{operation_type="list_containers"} 869716
kubelet_docker_operations_latency_microseconds{operation_type="list_images",quantile="0.5"} 79959
kubelet_docker_operations_latency_microseconds{operation_type="list_images",quantile="0.9"} 94146
kubelet_docker_operations_latency_microseconds{operation_type="list_images",quantile="0.99"} 114361
kubelet_docker_operations_latency_microseconds_sum{operation_type="list_images"} 2.724541692e+09
kubelet_docker_operations_latency_microseconds_count{operation_type="list_images"} 33350
kubelet_docker_operations_latency_microseconds{operation_type="pull_image",quantile="0.5"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="pull_image",quantile="0.9"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="pull_image",quantile="0.99"} NaN
kubelet_docker_operations_latency_microseconds_sum{operation_type="pull_image"} 1.88765137e+08
kubelet_docker_operations_latency_microseconds_count{operation_type="pull_image"} 19
kubelet_docker_operations_latency_microseconds{operation_type="remove_container",quantile="0.5"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="remove_container",quantile="0.9"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="remove_container",quantile="0.99"} NaN
kubelet_docker_operations_latency_microseconds_sum{operation_type="remove_container"} 1.973908e+06
kubelet_docker_operations_latency_microseconds_count{operation_type="remove_container"} 114
kubelet_docker_operations_latency_microseconds{operation_type="start_container",quantile="0.5"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="start_container",quantile="0.9"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="start_container",quantile="0.99"} NaN
kubelet_docker_operations_latency_microseconds_sum{operation_type="start_container"} 5.2653478e+07
kubelet_docker_operations_latency_microseconds_count{operation_type="start_container"} 111
kubelet_docker_operations_latency_microseconds{operation_type="stop_container",quantile="0.5"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="stop_container",quantile="0.9"} NaN
kubelet_docker_operations_latency_microseconds{operation_type="stop_container",quantile="0.99"} NaN
kubelet_docker_operations_latency_microseconds_sum{operation_type="stop_container"} 7.10673458e+08
kubelet_docker_operations_latency_microseconds_count{operation_type="stop_container"} 187
kubelet_docker_operations_latency_microseconds{operation_type="version",quantile="0.5"} 709
kubelet_docker_operations_latency_microseconds{operation_type="version",quantile="0.9"} 781
kubelet_docker_operations_latency_microseconds{operation_type="version",quantile="0.99"} 871
kubelet_docker_operations_latency_microseconds_sum{operation_type="version"} 4.7936272e+07
kubelet_docker_operations_latency_microseconds_count{operation_type="version"} 70559
# HELP kubelet_docker_operations_total Cumulative number of Docker operations by operation type.
# TYPE kubelet_docker_operations_total counter
kubelet_docker_operations_total{operation_type="create_container"} 111
kubelet_docker_operations_total{operation_type="info"} 2
kubelet_docker_operations_total{operation_type="inspect_container"} 1272
kubelet_docker_operations_total{operation_type="inspect_image"} 1432
kubelet_docker_operations_total{operation_type="list_containers"} 869716
kubelet_docker_operations_total{operation_type="list_images"} 33350
kubelet_docker_operations_total{operation_type="pull_image"} 19
kubelet_docker_operations_total{operation_type="remove_container"} 114
kubelet_docker_operations_total{operation_type="start_container"} 111
kubelet_docker_operations_total{operation_type="stop_container"} 187
kubelet_docker_operations_total{operation_type="version"} 70559
# HELP kubelet_network_plugin_operations_duration_seconds Latency in seconds of network plugin operations. Broken down by operation type.
# TYPE kubelet_network_plugin_operations_duration_seconds histogram
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="0.005"} 123
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="0.01"} 136
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="0.025"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="0.05"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="0.1"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="0.25"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="0.5"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="1"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="2.5"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="5"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="10"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="get_pod_network_status",le="+Inf"} 143
kubelet_network_plugin_operations_duration_seconds_sum{operation_type="get_pod_network_status"} 0.5713834720000002
kubelet_network_plugin_operations_duration_seconds_count{operation_type="get_pod_network_status"} 143
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="0.005"} 0
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="0.01"} 0
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="0.025"} 0
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="0.05"} 23
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="0.1"} 26
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="0.25"} 26
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="0.5"} 27
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="1"} 27
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="2.5"} 27
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="5"} 27
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="10"} 27
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="set_up_pod",le="+Inf"} 27
kubelet_network_plugin_operations_duration_seconds_sum{operation_type="set_up_pod"} 1.2514652690000003
kubelet_network_plugin_operations_duration_seconds_count{operation_type="set_up_pod"} 27
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="0.005"} 1
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="0.01"} 2
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="0.025"} 3
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="0.05"} 25
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="0.1"} 28
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="0.25"} 28
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="0.5"} 28
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="1"} 28
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="2.5"} 28
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="5"} 28
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="10"} 28
kubelet_network_plugin_operations_duration_seconds_bucket{operation_type="tear_down_pod",le="+Inf"} 28
kubelet_network_plugin_operations_duration_seconds_sum{operation_type="tear_down_pod"} 1.126804529
kubelet_network_plugin_operations_duration_seconds_count{operation_type="tear_down_pod"} 28
# HELP kubelet_network_plugin_operations_latency_microseconds (Deprecated) Latency in microseconds of network plugin operations. Broken down by operation type.
# TYPE kubelet_network_plugin_operations_latency_microseconds summary
kubelet_network_plugin_operations_latency_microseconds{operation_type="get_pod_network_status",quantile="0.5"} NaN
kubelet_network_plugin_operations_latency_microseconds{operation_type="get_pod_network_status",quantile="0.9"} NaN
kubelet_network_plugin_operations_latency_microseconds{operation_type="get_pod_network_status",quantile="0.99"} NaN
kubelet_network_plugin_operations_latency_microseconds_sum{operation_type="get_pod_network_status"} 571661
kubelet_network_plugin_operations_latency_microseconds_count{operation_type="get_pod_network_status"} 143
kubelet_network_plugin_operations_latency_microseconds{operation_type="set_up_pod",quantile="0.5"} NaN
kubelet_network_plugin_operations_latency_microseconds{operation_type="set_up_pod",quantile="0.9"} NaN
kubelet_network_plugin_operations_latency_microseconds{operation_type="set_up_pod",quantile="0.99"} NaN
kubelet_network_plugin_operations_latency_microseconds_sum{operation_type="set_up_pod"} 1.251609e+06
kubelet_network_plugin_operations_latency_microseconds_count{operation_type="set_up_pod"} 27
kubelet_network_plugin_operations_latency_microseconds{operation_type="tear_down_pod",quantile="0.5"} NaN
kubelet_network_plugin_operations_latency_microseconds{operation_type="tear_down_pod",quantile="0.9"} NaN
kubelet_network_plugin_operations_latency_microseconds{operation_type="tear_down_pod",quantile="0.99"} NaN
kubelet_network_plugin_operations_latency_microseconds_sum{operation_type="tear_down_pod"} 1.126963e+06
kubelet_network_plugin_operations_latency_microseconds_count{operation_type="tear_down_pod"} 28
# HELP kubelet_node_config_error This metric is true (1) if the node is experiencing a configuration-related error, false (0) otherwise.
# TYPE kubelet_node_config_error gauge
kubelet_node_config_error 0
# HELP kubelet_node_name The node's name. The count is always 1.
# TYPE kubelet_node_name gauge
kubelet_node_name{node="cn-shenzhen.i-wz92d8u10y3fu7gyu74n"} 1
# HELP kubelet_pleg_relist_duration_seconds Duration in seconds for relisting pods in PLEG.
# TYPE kubelet_pleg_relist_duration_seconds histogram
kubelet_pleg_relist_duration_seconds_bucket{le="0.005"} 0
kubelet_pleg_relist_duration_seconds_bucket{le="0.01"} 0
kubelet_pleg_relist_duration_seconds_bucket{le="0.025"} 225847
kubelet_pleg_relist_duration_seconds_bucket{le="0.05"} 231474
kubelet_pleg_relist_duration_seconds_bucket{le="0.1"} 231596
kubelet_pleg_relist_duration_seconds_bucket{le="0.25"} 231612
kubelet_pleg_relist_duration_seconds_bucket{le="0.5"} 231628
kubelet_pleg_relist_duration_seconds_bucket{le="1"} 231632
kubelet_pleg_relist_duration_seconds_bucket{le="2.5"} 231635
kubelet_pleg_relist_duration_seconds_bucket{le="5"} 231635
kubelet_pleg_relist_duration_seconds_bucket{le="10"} 231635
kubelet_pleg_relist_duration_seconds_bucket{le="+Inf"} 231635
kubelet_pleg_relist_duration_seconds_sum 3541.6588414329785
kubelet_pleg_relist_duration_seconds_count 231635
# HELP kubelet_pleg_relist_interval_microseconds (Deprecated) Interval in microseconds between relisting in PLEG.
# TYPE kubelet_pleg_relist_interval_microseconds summary
kubelet_pleg_relist_interval_microseconds{quantile="0.5"} 1.015301e+06
kubelet_pleg_relist_interval_microseconds{quantile="0.9"} 1.02135e+06
kubelet_pleg_relist_interval_microseconds{quantile="0.99"} 1.029384e+06
kubelet_pleg_relist_interval_microseconds_sum 2.35214359102e+11
kubelet_pleg_relist_interval_microseconds_count 231634
# HELP kubelet_pleg_relist_interval_seconds Interval in seconds between relisting in PLEG.
# TYPE kubelet_pleg_relist_interval_seconds histogram
kubelet_pleg_relist_interval_seconds_bucket{le="0.005"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.01"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.025"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.05"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.1"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.25"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.5"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="1"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="2.5"} 231631
kubelet_pleg_relist_interval_seconds_bucket{le="5"} 231634
kubelet_pleg_relist_interval_seconds_bucket{le="10"} 231634
kubelet_pleg_relist_interval_seconds_bucket{le="+Inf"} 231634
kubelet_pleg_relist_interval_seconds_sum 235213.94654464128
kubelet_pleg_relist_interval_seconds_count 231634
# HELP kubelet_pleg_relist_latency_microseconds (Deprecated) Latency in microseconds for relisting pods in PLEG.
# TYPE kubelet_pleg_relist_latency_microseconds summary
kubelet_pleg_relist_latency_microseconds{quantile="0.5"} 15123
kubelet_pleg_relist_latency_microseconds{quantile="0.9"} 21146
kubelet_pleg_relist_latency_microseconds{quantile="0.99"} 29315
kubelet_pleg_relist_latency_microseconds_sum 3.541714848e+09
kubelet_pleg_relist_latency_microseconds_count 231635
# HELP kubelet_pod_start_duration_seconds Duration in seconds for a single pod to go from pending to running.
# TYPE kubelet_pod_start_duration_seconds histogram
kubelet_pod_start_duration_seconds_bucket{le="0.005"} 5
kubelet_pod_start_duration_seconds_bucket{le="0.01"} 24
kubelet_pod_start_duration_seconds_bucket{le="0.025"} 33
kubelet_pod_start_duration_seconds_bucket{le="0.05"} 33
kubelet_pod_start_duration_seconds_bucket{le="0.1"} 33
kubelet_pod_start_duration_seconds_bucket{le="0.25"} 33
kubelet_pod_start_duration_seconds_bucket{le="0.5"} 33
kubelet_pod_start_duration_seconds_bucket{le="1"} 33
kubelet_pod_start_duration_seconds_bucket{le="2.5"} 39
kubelet_pod_start_duration_seconds_bucket{le="5"} 68
kubelet_pod_start_duration_seconds_bucket{le="10"} 72
kubelet_pod_start_duration_seconds_bucket{le="+Inf"} 81
kubelet_pod_start_duration_seconds_sum 330.1367666919998
kubelet_pod_start_duration_seconds_count 81
# HELP kubelet_pod_start_latency_microseconds (Deprecated) Latency in microseconds for a single pod to go from pending to running.
# TYPE kubelet_pod_start_latency_microseconds summary
kubelet_pod_start_latency_microseconds{quantile="0.5"} NaN
kubelet_pod_start_latency_microseconds{quantile="0.9"} NaN
kubelet_pod_start_latency_microseconds{quantile="0.99"} NaN
kubelet_pod_start_latency_microseconds_sum 3.30136806e+08
kubelet_pod_start_latency_microseconds_count 81
# HELP kubelet_pod_worker_duration_seconds Duration in seconds to sync a single pod. Broken down by operation type: create, update, or sync
# TYPE kubelet_pod_worker_duration_seconds histogram
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="0.005"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="0.01"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="0.025"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="0.05"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="0.1"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="0.25"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="0.5"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="1"} 14
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="2.5"} 15
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="5"} 15
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="10"} 16
kubelet_pod_worker_duration_seconds_bucket{operation_type="sync",le="+Inf"} 17
kubelet_pod_worker_duration_seconds_sum{operation_type="sync"} 22.843940947
kubelet_pod_worker_duration_seconds_count{operation_type="sync"} 17
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="0.005"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="0.01"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="0.025"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="0.05"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="0.1"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="0.25"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="0.5"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="1"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="2.5"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="5"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="10"} 0
kubelet_pod_worker_duration_seconds_bucket{operation_type="update",le="+Inf"} 1
kubelet_pod_worker_duration_seconds_sum{operation_type="update"} 30.689857875
kubelet_pod_worker_duration_seconds_count{operation_type="update"} 1
# HELP kubelet_pod_worker_latency_microseconds (Deprecated) Latency in microseconds to sync a single pod. Broken down by operation type: create, update, or sync
# TYPE kubelet_pod_worker_latency_microseconds summary
kubelet_pod_worker_latency_microseconds{operation_type="sync",quantile="0.5"} NaN
kubelet_pod_worker_latency_microseconds{operation_type="sync",quantile="0.9"} NaN
kubelet_pod_worker_latency_microseconds{operation_type="sync",quantile="0.99"} NaN
kubelet_pod_worker_latency_microseconds_sum{operation_type="sync"} 2.2844024e+07
kubelet_pod_worker_latency_microseconds_count{operation_type="sync"} 17
kubelet_pod_worker_latency_microseconds{operation_type="update",quantile="0.5"} NaN
kubelet_pod_worker_latency_microseconds{operation_type="update",quantile="0.9"} NaN
kubelet_pod_worker_latency_microseconds{operation_type="update",quantile="0.99"} NaN
kubelet_pod_worker_latency_microseconds_sum{operation_type="update"} 3.0689894e+07
kubelet_pod_worker_latency_microseconds_count{operation_type="update"} 1
# HELP kubelet_pod_worker_start_duration_seconds Duration in seconds from seeing a pod to starting a worker.
# TYPE kubelet_pod_worker_start_duration_seconds histogram
kubelet_pod_worker_start_duration_seconds_bucket{le="0.005"} 5
kubelet_pod_worker_start_duration_seconds_bucket{le="0.01"} 24
kubelet_pod_worker_start_duration_seconds_bucket{le="0.025"} 33
kubelet_pod_worker_start_duration_seconds_bucket{le="0.05"} 33
kubelet_pod_worker_start_duration_seconds_bucket{le="0.1"} 33
kubelet_pod_worker_start_duration_seconds_bucket{le="0.25"} 33
kubelet_pod_worker_start_duration_seconds_bucket{le="0.5"} 33
kubelet_pod_worker_start_duration_seconds_bucket{le="1"} 33
kubelet_pod_worker_start_duration_seconds_bucket{le="2.5"} 33
kubelet_pod_worker_start_duration_seconds_bucket{le="5"} 49
kubelet_pod_worker_start_duration_seconds_bucket{le="10"} 49
kubelet_pod_worker_start_duration_seconds_bucket{le="+Inf"} 49
kubelet_pod_worker_start_duration_seconds_sum 61.53770452699999
kubelet_pod_worker_start_duration_seconds_count 49
# HELP kubelet_pod_worker_start_latency_microseconds (Deprecated) Latency in microseconds from seeing a pod to starting a worker.
# TYPE kubelet_pod_worker_start_latency_microseconds summary
kubelet_pod_worker_start_latency_microseconds{quantile="0.5"} NaN
kubelet_pod_worker_start_latency_microseconds{quantile="0.9"} NaN
kubelet_pod_worker_start_latency_microseconds{quantile="0.99"} NaN
kubelet_pod_worker_start_latency_microseconds_sum 6.153773e+07
kubelet_pod_worker_start_latency_microseconds_count 49
# HELP kubelet_running_container_count Number of containers currently running
# TYPE kubelet_running_container_count gauge
kubelet_running_container_count 24
# HELP kubelet_running_pod_count Number of pods currently running
# TYPE kubelet_running_pod_count gauge
kubelet_running_pod_count 17
# HELP kubelet_runtime_operations (Deprecated) Cumulative number of runtime operations by operation type.
# TYPE kubelet_runtime_operations counter
kubelet_runtime_operations{operation_type="container_status"} 485
kubelet_runtime_operations{operation_type="create_container"} 78
kubelet_runtime_operations{operation_type="exec"} 1
kubelet_runtime_operations{operation_type="exec_sync"} 243056
kubelet_runtime_operations{operation_type="image_status"} 941
kubelet_runtime_operations{operation_type="list_containers"} 436800
kubelet_runtime_operations{operation_type="list_images"} 33350
kubelet_runtime_operations{operation_type="list_podsandbox"} 432881
kubelet_runtime_operations{operation_type="podsandbox_status"} 262
kubelet_runtime_operations{operation_type="pull_image"} 19
kubelet_runtime_operations{operation_type="remove_container"} 79
kubelet_runtime_operations{operation_type="run_podsandbox"} 33
kubelet_runtime_operations{operation_type="start_container"} 78
kubelet_runtime_operations{operation_type="status"} 47032
kubelet_runtime_operations{operation_type="stop_container"} 51
kubelet_runtime_operations{operation_type="stop_podsandbox"} 101
kubelet_runtime_operations{operation_type="update_runtime_config"} 1
kubelet_runtime_operations{operation_type="version"} 23495
# HELP kubelet_runtime_operations_duration_seconds Duration in seconds of runtime operations. Broken down by operation type.
# TYPE kubelet_runtime_operations_duration_seconds histogram
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.005"} 426
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.01"} 450
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.025"} 461
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.05"} 464
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.1"} 469
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.25"} 477
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.5"} 483
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="1"} 484
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="2.5"} 485
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="5"} 485
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="10"} 485
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="+Inf"} 485
kubelet_runtime_operations_duration_seconds_sum{operation_type="container_status"} 7.843776627999998
kubelet_runtime_operations_duration_seconds_count{operation_type="container_status"} 485
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.01"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.025"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.05"} 60
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.1"} 77
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.25"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.5"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="1"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="2.5"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="5"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="10"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="+Inf"} 78
kubelet_runtime_operations_duration_seconds_sum{operation_type="create_container"} 3.4917352999999993
kubelet_runtime_operations_duration_seconds_count{operation_type="create_container"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="0.005"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="0.01"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="0.025"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="0.05"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="0.1"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="0.25"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="0.5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="1"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="2.5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="10"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec",le="+Inf"} 1
kubelet_runtime_operations_duration_seconds_sum{operation_type="exec"} 0.004048766
kubelet_runtime_operations_duration_seconds_count{operation_type="exec"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="0.01"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="0.025"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="0.05"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="0.1"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="0.25"} 206080
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="0.5"} 243035
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="1"} 243049
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="2.5"} 243055
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="5"} 243056
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="10"} 243056
kubelet_runtime_operations_duration_seconds_bucket{operation_type="exec_sync",le="+Inf"} 243056
kubelet_runtime_operations_duration_seconds_sum{operation_type="exec_sync"} 56849.58025321542
kubelet_runtime_operations_duration_seconds_count{operation_type="exec_sync"} 243056
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.005"} 927
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.01"} 936
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.025"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.05"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.1"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.25"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.5"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="1"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="2.5"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="5"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="10"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="+Inf"} 941
kubelet_runtime_operations_duration_seconds_sum{operation_type="image_status"} 1.734655472
kubelet_runtime_operations_duration_seconds_count{operation_type="image_status"} 941
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.005"} 218
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.01"} 379889
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.025"} 435216
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.05"} 436780
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.1"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.25"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.5"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="1"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="2.5"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="5"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="10"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="+Inf"} 436800
kubelet_runtime_operations_duration_seconds_sum{operation_type="list_containers"} 3552.4815918728823
kubelet_runtime_operations_duration_seconds_count{operation_type="list_containers"} 436800
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.01"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.025"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.05"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.1"} 29993
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.25"} 33350
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.5"} 33350
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="1"} 33350
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="2.5"} 33350
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="5"} 33350
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="10"} 33350
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="+Inf"} 33350
kubelet_runtime_operations_duration_seconds_sum{operation_type="list_images"} 2752.961591391012
kubelet_runtime_operations_duration_seconds_count{operation_type="list_images"} 33350
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.005"} 132450
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.01"} 417841
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.025"} 432569
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.05"} 432874
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.1"} 432878
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.25"} 432878
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.5"} 432879
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="1"} 432880
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="2.5"} 432881
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="5"} 432881
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="10"} 432881
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="+Inf"} 432881
kubelet_runtime_operations_duration_seconds_sum{operation_type="list_podsandbox"} 2502.107261991974
kubelet_runtime_operations_duration_seconds_count{operation_type="list_podsandbox"} 432881
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.005"} 195
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.01"} 227
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.025"} 240
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.05"} 241
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.1"} 248
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.25"} 254
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.5"} 258
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="1"} 261
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="2.5"} 262
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="5"} 262
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="10"} 262
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="+Inf"} 262
kubelet_runtime_operations_duration_seconds_sum{operation_type="podsandbox_status"} 7.704051059000002
kubelet_runtime_operations_duration_seconds_count{operation_type="podsandbox_status"} 262
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.01"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.025"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.05"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.1"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.25"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="1"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="2.5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="5"} 7
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="10"} 10
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="+Inf"} 19
kubelet_runtime_operations_duration_seconds_sum{operation_type="pull_image"} 188.798971836
kubelet_runtime_operations_duration_seconds_count{operation_type="pull_image"} 19
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="0.01"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="0.025"} 63
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="0.05"} 77
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="0.1"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="0.25"} 79
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="0.5"} 79
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="1"} 79
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="2.5"} 79
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="5"} 79
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="10"} 79
kubelet_runtime_operations_duration_seconds_bucket{operation_type="remove_container",le="+Inf"} 79
kubelet_runtime_operations_duration_seconds_sum{operation_type="remove_container"} 1.5799119969999995
kubelet_runtime_operations_duration_seconds_count{operation_type="remove_container"} 79
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.01"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.025"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.05"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.1"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.25"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.5"} 6
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="1"} 32
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="2.5"} 33
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="5"} 33
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="10"} 33
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="+Inf"} 33
kubelet_runtime_operations_duration_seconds_sum{operation_type="run_podsandbox"} 19.390970901000006
kubelet_runtime_operations_duration_seconds_count{operation_type="run_podsandbox"} 33
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.01"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.025"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.05"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.1"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.25"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.5"} 56
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="1"} 77
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="2.5"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="5"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="10"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="+Inf"} 78
kubelet_runtime_operations_duration_seconds_sum{operation_type="start_container"} 36.316746111
kubelet_runtime_operations_duration_seconds_count{operation_type="start_container"} 78
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.005"} 46837
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.01"} 47006
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.025"} 47031
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.05"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.1"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.25"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.5"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="1"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="2.5"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="5"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="10"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="+Inf"} 47032
kubelet_runtime_operations_duration_seconds_sum{operation_type="status"} 54.765071607000074
kubelet_runtime_operations_duration_seconds_count{operation_type="status"} 47032
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="0.005"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="0.01"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="0.025"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="0.05"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="0.1"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="0.25"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="0.5"} 9
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="1"} 9
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="2.5"} 9
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="5"} 10
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="10"} 32
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_container",le="+Inf"} 51
kubelet_runtime_operations_duration_seconds_sum{operation_type="stop_container"} 698.493260831
kubelet_runtime_operations_duration_seconds_count{operation_type="stop_container"} 51
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="0.005"} 68
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="0.01"} 69
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="0.025"} 69
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="0.05"} 69
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="0.1"} 69
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="0.25"} 69
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="0.5"} 100
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="1"} 100
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="2.5"} 100
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="5"} 101
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="10"} 101
kubelet_runtime_operations_duration_seconds_bucket{operation_type="stop_podsandbox",le="+Inf"} 101
kubelet_runtime_operations_duration_seconds_sum{operation_type="stop_podsandbox"} 13.427133009
kubelet_runtime_operations_duration_seconds_count{operation_type="stop_podsandbox"} 101
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="0.005"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="0.01"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="0.025"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="0.05"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="0.1"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="0.25"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="0.5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="1"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="2.5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="5"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="10"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="update_runtime_config",le="+Inf"} 1
kubelet_runtime_operations_duration_seconds_sum{operation_type="update_runtime_config"} 0.000937003
kubelet_runtime_operations_duration_seconds_count{operation_type="update_runtime_config"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.005"} 23450
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.01"} 23490
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.025"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.05"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.1"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.25"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.5"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="1"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="2.5"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="5"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="10"} 23495
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="+Inf"} 23495
kubelet_runtime_operations_duration_seconds_sum{operation_type="version"} 18.979898943999967
kubelet_runtime_operations_duration_seconds_count{operation_type="version"} 23495
# HELP kubelet_runtime_operations_errors (Deprecated) Cumulative number of runtime operation errors by operation type.
# TYPE kubelet_runtime_operations_errors counter
kubelet_runtime_operations_errors{operation_type="container_status"} 46
kubelet_runtime_operations_errors{operation_type="stop_podsandbox"} 1
# HELP kubelet_runtime_operations_errors_total Cumulative number of runtime operation errors by operation type.
# TYPE kubelet_runtime_operations_errors_total counter
kubelet_runtime_operations_errors_total{operation_type="container_status"} 46
kubelet_runtime_operations_errors_total{operation_type="stop_podsandbox"} 1
# HELP kubelet_runtime_operations_latency_microseconds (Deprecated) Latency in microseconds of runtime operations. Broken down by operation type.
# TYPE kubelet_runtime_operations_latency_microseconds summary
kubelet_runtime_operations_latency_microseconds{operation_type="container_status",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="container_status",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="container_status",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="container_status"} 7.844293e+06
kubelet_runtime_operations_latency_microseconds_count{operation_type="container_status"} 485
kubelet_runtime_operations_latency_microseconds{operation_type="create_container",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="create_container",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="create_container",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="create_container"} 3.491939e+06
kubelet_runtime_operations_latency_microseconds_count{operation_type="create_container"} 78
kubelet_runtime_operations_latency_microseconds{operation_type="exec",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="exec",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="exec",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="exec"} 4080
kubelet_runtime_operations_latency_microseconds_count{operation_type="exec"} 1
kubelet_runtime_operations_latency_microseconds{operation_type="exec_sync",quantile="0.5"} 231179
kubelet_runtime_operations_latency_microseconds{operation_type="exec_sync",quantile="0.9"} 276101
kubelet_runtime_operations_latency_microseconds{operation_type="exec_sync",quantile="0.99"} 324786
kubelet_runtime_operations_latency_microseconds_sum{operation_type="exec_sync"} 5.6850281846e+10
kubelet_runtime_operations_latency_microseconds_count{operation_type="exec_sync"} 243056
kubelet_runtime_operations_latency_microseconds{operation_type="image_status",quantile="0.5"} 1735
kubelet_runtime_operations_latency_microseconds{operation_type="image_status",quantile="0.9"} 1735
kubelet_runtime_operations_latency_microseconds{operation_type="image_status",quantile="0.99"} 1735
kubelet_runtime_operations_latency_microseconds_sum{operation_type="image_status"} 1.736268e+06
kubelet_runtime_operations_latency_microseconds_count{operation_type="image_status"} 941
kubelet_runtime_operations_latency_microseconds{operation_type="list_containers",quantile="0.5"} 8289
kubelet_runtime_operations_latency_microseconds{operation_type="list_containers",quantile="0.9"} 11665
kubelet_runtime_operations_latency_microseconds{operation_type="list_containers",quantile="0.99"} 19521
kubelet_runtime_operations_latency_microseconds_sum{operation_type="list_containers"} 3.553153405e+09
kubelet_runtime_operations_latency_microseconds_count{operation_type="list_containers"} 436800
kubelet_runtime_operations_latency_microseconds{operation_type="list_images",quantile="0.5"} 80697
kubelet_runtime_operations_latency_microseconds{operation_type="list_images",quantile="0.9"} 95129
kubelet_runtime_operations_latency_microseconds{operation_type="list_images",quantile="0.99"} 115076
kubelet_runtime_operations_latency_microseconds_sum{operation_type="list_images"} 2.753022618e+09
kubelet_runtime_operations_latency_microseconds_count{operation_type="list_images"} 33350
kubelet_runtime_operations_latency_microseconds{operation_type="list_podsandbox",quantile="0.5"} 5803
kubelet_runtime_operations_latency_microseconds{operation_type="list_podsandbox",quantile="0.9"} 7658
kubelet_runtime_operations_latency_microseconds{operation_type="list_podsandbox",quantile="0.99"} 14387
kubelet_runtime_operations_latency_microseconds_sum{operation_type="list_podsandbox"} 2.502804091e+09
kubelet_runtime_operations_latency_microseconds_count{operation_type="list_podsandbox"} 432881
kubelet_runtime_operations_latency_microseconds{operation_type="podsandbox_status",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="podsandbox_status",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="podsandbox_status",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="podsandbox_status"} 7.70447e+06
kubelet_runtime_operations_latency_microseconds_count{operation_type="podsandbox_status"} 262
kubelet_runtime_operations_latency_microseconds{operation_type="pull_image",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="pull_image",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="pull_image",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="pull_image"} 1.88799027e+08
kubelet_runtime_operations_latency_microseconds_count{operation_type="pull_image"} 19
kubelet_runtime_operations_latency_microseconds{operation_type="remove_container",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="remove_container",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="remove_container",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="remove_container"} 1.580078e+06
kubelet_runtime_operations_latency_microseconds_count{operation_type="remove_container"} 79
kubelet_runtime_operations_latency_microseconds{operation_type="run_podsandbox",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="run_podsandbox",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="run_podsandbox",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="run_podsandbox"} 1.9391053e+07
kubelet_runtime_operations_latency_microseconds_count{operation_type="run_podsandbox"} 33
kubelet_runtime_operations_latency_microseconds{operation_type="start_container",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="start_container",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="start_container",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="start_container"} 3.6316928e+07
kubelet_runtime_operations_latency_microseconds_count{operation_type="start_container"} 78
kubelet_runtime_operations_latency_microseconds{operation_type="status",quantile="0.5"} 1136
kubelet_runtime_operations_latency_microseconds{operation_type="status",quantile="0.9"} 1231
kubelet_runtime_operations_latency_microseconds{operation_type="status",quantile="0.99"} 1374
kubelet_runtime_operations_latency_microseconds_sum{operation_type="status"} 5.4844184e+07
kubelet_runtime_operations_latency_microseconds_count{operation_type="status"} 47032
kubelet_runtime_operations_latency_microseconds{operation_type="stop_container",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="stop_container",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="stop_container",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="stop_container"} 6.98493363e+08
kubelet_runtime_operations_latency_microseconds_count{operation_type="stop_container"} 51
kubelet_runtime_operations_latency_microseconds{operation_type="stop_podsandbox",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="stop_podsandbox",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="stop_podsandbox",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="stop_podsandbox"} 1.3427308e+07
kubelet_runtime_operations_latency_microseconds_count{operation_type="stop_podsandbox"} 101
kubelet_runtime_operations_latency_microseconds{operation_type="update_runtime_config",quantile="0.5"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="update_runtime_config",quantile="0.9"} NaN
kubelet_runtime_operations_latency_microseconds{operation_type="update_runtime_config",quantile="0.99"} NaN
kubelet_runtime_operations_latency_microseconds_sum{operation_type="update_runtime_config"} 989
kubelet_runtime_operations_latency_microseconds_count{operation_type="update_runtime_config"} 1
kubelet_runtime_operations_latency_microseconds{operation_type="version",quantile="0.5"} 766
kubelet_runtime_operations_latency_microseconds{operation_type="version",quantile="0.9"} 832
kubelet_runtime_operations_latency_microseconds{operation_type="version",quantile="0.99"} 1273
kubelet_runtime_operations_latency_microseconds_sum{operation_type="version"} 1.9018191e+07
kubelet_runtime_operations_latency_microseconds_count{operation_type="version"} 23495
# HELP kubelet_runtime_operations_total Cumulative number of runtime operations by operation type.
# TYPE kubelet_runtime_operations_total counter
kubelet_runtime_operations_total{operation_type="container_status"} 485
kubelet_runtime_operations_total{operation_type="create_container"} 78
kubelet_runtime_operations_total{operation_type="exec"} 1
kubelet_runtime_operations_total{operation_type="exec_sync"} 243056
kubelet_runtime_operations_total{operation_type="image_status"} 941
kubelet_runtime_operations_total{operation_type="list_containers"} 436800
kubelet_runtime_operations_total{operation_type="list_images"} 33350
kubelet_runtime_operations_total{operation_type="list_podsandbox"} 432881
kubelet_runtime_operations_total{operation_type="podsandbox_status"} 262
kubelet_runtime_operations_total{operation_type="pull_image"} 19
kubelet_runtime_operations_total{operation_type="remove_container"} 79
kubelet_runtime_operations_total{operation_type="run_podsandbox"} 33
kubelet_runtime_operations_total{operation_type="start_container"} 78
kubelet_runtime_operations_total{operation_type="status"} 47032
kubelet_runtime_operations_total{operation_type="stop_container"} 51
kubelet_runtime_operations_total{operation_type="stop_podsandbox"} 101
kubelet_runtime_operations_total{operation_type="update_runtime_config"} 1
kubelet_runtime_operations_total{operation_type="version"} 23495
# HELP kubernetes_build_info A metric with a constant '1' value labeled by major, minor, git version, git commit, git tree state, build date, Go version, and compiler from which Kubernetes was built, and platform on which it is running.
# TYPE kubernetes_build_info gauge
kubernetes_build_info{buildDate="2019-10-16T08:31:01Z",compiler="gc",gitCommit="51888f5",gitTreeState="",gitVersion="v1.14.8-aliyun.1",goVersion="go1.12.10",major="1",minor="14+",platform="linux/amd64"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 17487.03
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 159
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.28847616e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.57598307735e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.892851712e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP rest_client_request_duration_seconds Request latency in seconds. Broken down by verb and URL.
# TYPE rest_client_request_duration_seconds histogram
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.032"} 29
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.064"} 29
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.128"} 29
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.256"} 29
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.512"} 29
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="+Inf"} 29
rest_client_request_duration_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE"} 0.614138223
rest_client_request_duration_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE"} 29
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.004"} 11776
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.008"} 51497
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.016"} 55680
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.032"} 55803
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.064"} 55834
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.128"} 55837
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.256"} 55837
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.512"} 55837
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="+Inf"} 55838
rest_client_request_duration_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET"} 317.6928347449978
rest_client_request_duration_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET"} 55838
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.008"} 28
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.016"} 9621
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.032"} 10304
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.064"} 10340
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.128"} 10341
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.256"} 10342
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.512"} 10344
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="+Inf"} 10345
rest_client_request_duration_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH"} 134.7969249319997
rest_client_request_duration_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH"} 10345
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.004"} 7
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.008"} 217
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.016"} 392
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.032"} 395
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.064"} 395
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.128"} 395
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.256"} 395
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.512"} 395
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="+Inf"} 395
rest_client_request_duration_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST"} 3.2997518890000013
rest_client_request_duration_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST"} 395
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.008"} 9685
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.016"} 23358
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.032"} 23467
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.064"} 23483
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.128"} 23483
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.256"} 23484
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.512"} 23484
rest_client_request_duration_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="+Inf"} 23484
rest_client_request_duration_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT"} 196.64431807500105
rest_client_request_duration_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT"} 23484
# HELP rest_client_request_latency_seconds (Deprecated) Request latency in seconds. Broken down by verb and URL.
# TYPE rest_client_request_latency_seconds histogram
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.001"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.002"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.004"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.008"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.016"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.032"} 29
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.064"} 29
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.128"} 29
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.256"} 29
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="0.512"} 29
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE",le="+Inf"} 29
rest_client_request_latency_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE"} 0.614138223
rest_client_request_latency_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="DELETE"} 29
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.001"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.002"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.004"} 11776
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.008"} 51497
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.016"} 55680
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.032"} 55803
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.064"} 55834
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.128"} 55837
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.256"} 55837
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="0.512"} 55837
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET",le="+Inf"} 55838
rest_client_request_latency_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET"} 317.6928347449978
rest_client_request_latency_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="GET"} 55838
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.001"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.002"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.004"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.008"} 28
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.016"} 9621
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.032"} 10304
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.064"} 10340
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.128"} 10341
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.256"} 10342
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="0.512"} 10344
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH",le="+Inf"} 10345
rest_client_request_latency_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH"} 134.7969249319997
rest_client_request_latency_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PATCH"} 10345
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.001"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.002"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.004"} 7
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.008"} 217
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.016"} 392
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.032"} 395
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.064"} 395
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.128"} 395
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.256"} 395
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="0.512"} 395
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST",le="+Inf"} 395
rest_client_request_latency_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST"} 3.2997518890000013
rest_client_request_latency_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="POST"} 395
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.001"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.002"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.004"} 0
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.008"} 9685
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.016"} 23358
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.032"} 23467
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.064"} 23483
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.128"} 23483
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.256"} 23484
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="0.512"} 23484
rest_client_request_latency_seconds_bucket{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT",le="+Inf"} 23484
rest_client_request_latency_seconds_sum{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT"} 196.64431807500105
rest_client_request_latency_seconds_count{url="https://172.16.33.235:6443/%7Bprefix%7D",verb="PUT"} 23484
# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="172.16.33.235:6443",method="DELETE"} 29
rest_client_requests_total{code="200",host="172.16.33.235:6443",method="GET"} 76653
rest_client_requests_total{code="200",host="172.16.33.235:6443",method="PATCH"} 10340
rest_client_requests_total{code="200",host="172.16.33.235:6443",method="PUT"} 23484
rest_client_requests_total{code="201",host="172.16.33.235:6443",method="POST"} 394
rest_client_requests_total{code="404",host="172.16.33.235:6443",method="GET"} 33
rest_client_requests_total{code="404",host="172.16.33.235:6443",method="PATCH"} 5
rest_client_requests_total{code="409",host="172.16.33.235:6443",method="POST"} 1
# HELP storage_operation_duration_seconds Storage operation duration
# TYPE storage_operation_duration_seconds histogram
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="0.1"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="0.25"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="0.5"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="1"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="2.5"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="5"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="10"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="15"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="25"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="50"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="120"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="300"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="600"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap",le="+Inf"} 33
storage_operation_duration_seconds_sum{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap"} 0.014875406999999998
storage_operation_duration_seconds_count{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/configmap"} 33
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="0.1"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="0.25"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="0.5"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="1"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="2.5"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="5"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="10"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="15"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="25"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="50"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="120"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="300"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="600"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir",le="+Inf"} 26
storage_operation_duration_seconds_sum{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir"} 0.006441252999999999
storage_operation_duration_seconds_count{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/empty-dir"} 26
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="0.1"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="0.25"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="0.5"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="1"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="2.5"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="5"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="10"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="15"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="25"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="50"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="120"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="300"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="600"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path",le="+Inf"} 23
storage_operation_duration_seconds_sum{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path"} 0.020083386
storage_operation_duration_seconds_count{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/host-path"} 23
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="0.1"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="0.25"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="0.5"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="1"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="2.5"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="5"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="10"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="15"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="25"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="50"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="120"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="300"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="600"} 81
storage_operation_duration_seconds_bucket{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret",le="+Inf"} 81
storage_operation_duration_seconds_sum{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret"} 0.03433156099999999
storage_operation_duration_seconds_count{operation_name="verify_controller_attached_volume",volume_plugin="kubernetes.io/secret"} 81
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="0.1"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="0.25"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="0.5"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="1"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="2.5"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="5"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="10"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="15"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="25"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="50"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="120"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="300"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="600"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas",le="+Inf"} 1
storage_operation_duration_seconds_sum{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas"} 0.020981257
storage_operation_duration_seconds_count{operation_name="volume_attach",volume_plugin="flexvolume-alicloud/nas"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="0.1"} 0
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="0.25"} 0
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="0.5"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="1"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="2.5"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="5"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="10"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="15"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="25"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="50"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="120"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="300"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="600"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas",le="+Inf"} 1
storage_operation_duration_seconds_sum{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas"} 0.302120776
storage_operation_duration_seconds_count{operation_name="volume_mount",volume_plugin="flexvolume-alicloud/nas"} 1
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="0.1"} 27376
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="0.25"} 27376
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="0.5"} 27377
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="1"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="2.5"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="5"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="10"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="15"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="25"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="50"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="120"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="300"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="600"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap",le="+Inf"} 27378
storage_operation_duration_seconds_sum{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap"} 13.02840854100003
storage_operation_duration_seconds_count{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap"} 27378
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="0.1"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="0.25"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="0.5"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="1"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="2.5"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="5"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="10"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="15"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="25"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="50"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="120"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="300"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="600"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir",le="+Inf"} 26
storage_operation_duration_seconds_sum{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir"} 0.49128108000000004
storage_operation_duration_seconds_count{operation_name="volume_mount",volume_plugin="kubernetes.io/empty-dir"} 26
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="0.1"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="0.25"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="0.5"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="1"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="2.5"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="5"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="10"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="15"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="25"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="50"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="120"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="300"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="600"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path",le="+Inf"} 23
storage_operation_duration_seconds_sum{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path"} 0.047389741000000006
storage_operation_duration_seconds_count{operation_name="volume_mount",volume_plugin="kubernetes.io/host-path"} 23
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="0.1"} 71975
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="0.25"} 71976
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="0.5"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="1"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="2.5"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="5"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="10"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="15"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="25"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="50"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="120"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="300"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="600"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_mount",volume_plugin="kubernetes.io/secret",le="+Inf"} 71977
storage_operation_duration_seconds_sum{operation_name="volume_mount",volume_plugin="kubernetes.io/secret"} 33.26263116300012
storage_operation_duration_seconds_count{operation_name="volume_mount",volume_plugin="kubernetes.io/secret"} 71977
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="0.1"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="0.25"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="0.5"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="1"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="2.5"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="5"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="10"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="15"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="25"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="50"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="120"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="300"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="600"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap",le="+Inf"} 24
storage_operation_duration_seconds_sum{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap"} 0.016420714
storage_operation_duration_seconds_count{operation_name="volume_unmount",volume_plugin="kubernetes.io/configmap"} 24
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="0.1"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="0.25"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="0.5"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="1"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="2.5"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="5"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="10"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="15"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="25"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="50"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="120"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="300"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="600"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir",le="+Inf"} 22
storage_operation_duration_seconds_sum{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir"} 0.30775678999999995
storage_operation_duration_seconds_count{operation_name="volume_unmount",volume_plugin="kubernetes.io/empty-dir"} 22
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="0.1"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="0.25"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="0.5"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="1"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="2.5"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="5"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="10"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="15"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="25"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="50"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="120"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="300"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="600"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path",le="+Inf"} 6
storage_operation_duration_seconds_sum{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path"} 0.001087564
storage_operation_duration_seconds_count{operation_name="volume_unmount",volume_plugin="kubernetes.io/host-path"} 6
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="0.1"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="0.25"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="0.5"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="1"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="2.5"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="5"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="10"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="15"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="25"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="50"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="120"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="300"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="600"} 57
storage_operation_duration_seconds_bucket{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret",le="+Inf"} 57
storage_operation_duration_seconds_sum{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret"} 0.742680516
storage_operation_duration_seconds_count{operation_name="volume_unmount",volume_plugin="kubernetes.io/secret"} 57
# HELP storage_operation_errors_total Storage operation errors
# TYPE storage_operation_errors_total counter
storage_operation_errors_total{operation_name="volume_mount",volume_plugin="kubernetes.io/configmap"} 4
storage_operation_errors_total{operation_name="volume_mount",volume_plugin="kubernetes.io/secret"} 7
# HELP storage_operation_status_count Storage operation return statuses count
# TYPE storage_operation_status_count counter
storage_operation_status_count{operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/configmap"} 33
storage_operation_status_count{operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/empty-dir"} 26
storage_operation_status_count{operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/host-path"} 23
storage_operation_status_count{operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/secret"} 81
storage_operation_status_count{operation_name="volume_attach",status="success",volume_plugin="flexvolume-alicloud/nas"} 1
storage_operation_status_count{operation_name="volume_mount",status="fail-unknown",volume_plugin="kubernetes.io/configmap"} 4
storage_operation_status_count{operation_name="volume_mount",status="fail-unknown",volume_plugin="kubernetes.io/secret"} 7
storage_operation_status_count{operation_name="volume_mount",status="success",volume_plugin="flexvolume-alicloud/nas"} 1
storage_operation_status_count{operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/configmap"} 27378
storage_operation_status_count{operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/empty-dir"} 26
storage_operation_status_count{operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/host-path"} 23
storage_operation_status_count{operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/secret"} 71977
storage_operation_status_count{operation_name="volume_unmount",status="success",volume_plugin="kubernetes.io/configmap"} 24
storage_operation_status_count{operation_name="volume_unmount",status="success",volume_plugin="kubernetes.io/empty-dir"} 22
storage_operation_status_count{operation_name="volume_unmount",status="success",volume_plugin="kubernetes.io/host-path"} 6
storage_operation_status_count{operation_name="volume_unmount",status="success",volume_plugin="kubernetes.io/secret"} 57
# HELP volume_manager_total_volumes Number of volumes in Volume Manager
# TYPE volume_manager_total_volumes gauge
volume_manager_total_volumes{plugin_name="flexvolume-alicloud/nas",state="actual_state_of_world"} 1
volume_manager_total_volumes{plugin_name="flexvolume-alicloud/nas",state="desired_state_of_world"} 1
volume_manager_total_volumes{plugin_name="kubernetes.io/configmap",state="actual_state_of_world"} 9
volume_manager_total_volumes{plugin_name="kubernetes.io/configmap",state="desired_state_of_world"} 9
volume_manager_total_volumes{plugin_name="kubernetes.io/empty-dir",state="actual_state_of_world"} 4
volume_manager_total_volumes{plugin_name="kubernetes.io/empty-dir",state="desired_state_of_world"} 4
volume_manager_total_volumes{plugin_name="kubernetes.io/host-path",state="actual_state_of_world"} 17
volume_manager_total_volumes{plugin_name="kubernetes.io/host-path",state="desired_state_of_world"} 17
volume_manager_total_volumes{plugin_name="kubernetes.io/secret",state="actual_state_of_world"} 25
volume_manager_total_volumes{plugin_name="kubernetes.io/secret",state="desired_state_of_world"} 25

[cofyc]

do you have mounted volumes on that node?

[lianghuiyuan]

scrape nothing from kubelet-exporter url /metrics

$ kubectl exec -it centos-tool-auto-84c77c49bd-knhw5 -c centos -n lateinos  -- curl -X GET http://kubelet-exporter.monitoring.svc:9859/stats/summary
<html>
	<head>
		<title>Kube Metrics Server</title>
	</head>
	<body>
		<h1>Kube Metrics</h1>
		<ul>
			<li><a href='/metrics'>metrics</a></li>
			<li><a href='/healthz'>healthz</a></li>
		</ul>
	</body>
</html>

$ kubectl exec -it centos-tool-auto-84c77c49bd-knhw5 -c centos -n lateinos  -- curl -X GET http://kubelet-exporter.monitoring.svc:9859/metrics
(NOTHING)

[lianghuiyuan]

@cofyc
Think you!

Yes, I have mounted volumes.

 $ kubectl get pv 
NAME                                                                                           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                                 STORAGECLASS                      REASON   AGE
assisted-sentry-data-pv                                                                        100Gi      RWO            Retain           Bound       platform-assisted/assisted-sentry-data-pvc            nas                                        43d
assisted-sentry-postgres-pv                                                                    100Gi      RWO            Retain           Bound       platform-assisted/assisted-sentry-postgres-pvc        nas                                        43d
d-wz931c9n53r7osx95fx7                                                                         20Gi       RWO            Retain           Bound       istio-system/istio-grafana-pvc                        alicloud-disk-efficiency                   181d
jenkins-pv                                                                                     200Gi      RWX            Retain           Bound       cicd/jenkins-ci-pvc                                                                         196d
monitoring-alertmanager-main-db-alertmanager-main-0-pvc-3d0f0ef9-f236-11e9-91b0-aeb2e5e2ff0c   20Gi       RWO            Retain           Bound       monitoring/alertmanager-main-db-alertmanager-main-0   alertmanager-db-alicloud-nas-sc            55d
monitoring-alertmanager-main-db-alertmanager-main-1-pvc-3dde4f36-f236-11e9-91b0-aeb2e5e2ff0c   20Gi       RWO            Retain           Bound       monitoring/alertmanager-main-db-alertmanager-main-1   alertmanager-db-alicloud-nas-sc            55d
monitoring-alertmanager-main-db-alertmanager-main-2-pvc-3de1a501-f236-11e9-91b0-aeb2e5e2ff0c   20Gi       RWO            Retain           Bound       monitoring/alertmanager-main-db-alertmanager-main-2   alertmanager-db-alicloud-nas-sc            55d
monitoring-grafana-db-pvc-pvc-89aff0bc-1780-11ea-84bb-ae80a6d3a8ae                             10Gi       RWX            Retain           Bound       monitoring/grafana-db-pvc                             grafana-db-alicloud-nas-sc                 7d13h
monitoring-prometheus-k8s-db-prometheus-k8s-0-pvc-7fc402df-e060-11e9-91b0-aeb2e5e2ff0c         100Gi      RWO            Retain           Bound       monitoring/prometheus-k8s-db-prometheus-k8s-0         prometheus-db-alicloud-nas-sc              77d
monitoring-prometheus-k8s-db-prometheus-k8s-1-pvc-7fc6558c-e060-11e9-91b0-aeb2e5e2ff0c         100Gi      RWO            Retain           Bound       monitoring/prometheus-k8s-db-prometheus-k8s-1         prometheus-db-alicloud-nas-sc              77d
monitoring-prometheus-k8s-db-prometheus-k8s-2-pvc-77d6987c-0f30-11ea-91b0-aeb2e5e2ff0c         100Gi      RWO            Retain           Bound       monitoring/prometheus-k8s-db-prometheus-k8s-2         prometheus-db-alicloud-nas-sc              18d