Since the switch of the metrics backend the autoscaler is reporting stale metrics for services/revisions. We run:
metrics-protocol: prometheus
request-metrics-protocol: prometheus
Metrics of deleted revisions should no longer be reported.
Metrics of deleted revisions are reported with their last value.
Create a revision and delete it. These services/revisions all no longer exist and yet all metrics still report values like kn_revision_pods_count and kn_revision_pods_requested. All of which are stale:
kn_revision_pods_count{k8s_namespace_name="21uj92vxm973",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_count{k8s_namespace_name="21ujeqd9uia7",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_count{k8s_namespace_name="21ujiqdz3bpr",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_count{k8s_namespace_name="21uklm90kaa7",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_requested{k8s_namespace_name="21uh7qgrjzf3",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
kn_revision_pods_requested{k8s_namespace_name="21uhb52o4rnz",kn_configuration_name="my-app",kn_revision_name="my-app-00001",kn_service_name="my-app",otel_scope_name="knative.dev/serving/pkg/autoscaler",otel_scope_schema_url="",otel_scope_version=""} 1
Since the switch of the metrics backend the autoscaler is reporting stale metrics for services/revisions. We run:
What version of Knative?
v1.19.6
Expected Behavior
Metrics of deleted revisions should no longer be reported.
Actual Behavior
Metrics of deleted revisions are reported with their last value.
Steps to Reproduce the Problem
Create a revision and delete it. These services/revisions all no longer exist and yet all metrics still report values like
kn_revision_pods_countandkn_revision_pods_requested. All of which are stale: