What would you like to be added?

There are just to many ways metric reference resources, same label includes different contents causing confusion and making filtering hard and joining impossible.

Example metrics collected on a empty kind cluster by running kubectl get --raw /metrics | grep -i apiservice, it is not exhaustive:

• [recommended] {group="$GROUP", resource="$RESOURCE"} for example: {group="apiregistration.k8s.io",resource="apiservices"}

Metric	Stability	Example time series
apiserver_longrunning_requests	STABLE	{component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}
apiserver_request_duration_seconds	STABLE	{component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_sli_duration_seconds	ALPHA	{component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"}
apiserver_request_total	STABLE	{code="201",component="apiserver",dry_run="",group="apiregistration.k8s.io",resource="apiservices",scope="resource",subresource="",verb="POST",version="v1"} 20
apiserver_response_sizes	STABLE	{component="apiserver",group="apiregistration.k8s.io",resource="apiservices",scope="cluster",subresource="",verb="WATCH",version="v1"}

* Resource_prefix=”/${GROUP}/${RESOURCE}”, for example `{resource_prefix"/apiregistration.k8s.io/apiservices"}`

Metric	Stability	Example time series
apiserver_cache_list_fetched_objects_total	ALPHA	{index="",resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_cache_list_returned_objects_total	ALPHA	{resource_prefix="/apiregistration.k8s.io/apiservices"}
apiserver_cache_list_total	ALPHA	{index="",resource_prefix="/apiregistration.k8s.io/apiservices"}

• {resource="${RESOURCE}.${GROUP}"} for example {resource="apiservices.apiregistration.k8s.io"}

Metric	Stability	Example time series
apiserver_request_body_size_bytes	ALPHA	{resource="apiservices.apiregistration.k8s.io",verb="create"}
apiserver_storage_events_received_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_evaluated_objects_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_fetched_objects_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_returned_objects_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_list_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_storage_objects	STABLE	{resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_dispatched_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_events_received_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_initializations_total	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
apiserver_watch_cache_resource_version	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
etcd_bookmark_counts	ALPHA	{resource="apiservices.apiregistration.k8s.io"}
watch_cache_capacity	ALPHA	{resource="apiservices.apiregistration.k8s.io"}

• Just resource=”${RESOURCE}”, for example {resource="apiservices"}

Metric	Stability	Example time series
apiserver_selfrequest_total	ALPHA	{resource="apiservices",subresource="",verb="WATCH"}

• Type=”/registry/${GROUP}/${RESOURCE}”, for example {type="/registry/apiregistration.k8s.io/apiservices/"}

Metric	Stability	Example time series
etcd_request_duration_seconds	ALPHA	{operation="listWithCount",type="/registry/apiregistration.k8s.io/apiservices/"}
etcd_requests_total	ALPHA	{operation="listWithCount",type="/registry/apiregistration.k8s.io/apiservices/"}

• group=”${GROUP}”,kind=”${KIND}”, for example {group="",kind="ConfigMap"}

Metric	Stability	Example time series
apiserver_watch_events_sizes	ALPHA	{group="",kind="ConfigMap",version="v1"}
apiserver_watch_events_total	ALPHA	{group="",kind="ConfigMap",version="v1"}

Proposal

I propose to addopt {group="$GROUP", resource="$RESOURCE"} as we have multiple stable metrics already using this format. Most metrics are in ALPHA stage meaning we should be able to relabel them

Only exception is apiserver_storage_objects which uses {resource="${RESOURCE}.${GROUP}"}. As https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/1209-metrics-stability/kubernetes-control-plane-metrics-stability.md#stability-classes defines that adding a label is a breaking change, we would nee do deprecate it and replace with a new one.

Why is this needed?

Having predictable ways to find metrics relating to a resource
/sig instrumentation