![\"One](\"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/diagrams\/kardinalitaet-en.png\")
Bounded and Unbounded Labels<\/h2>\n
The useful distinction is not how many values a label has today but whether its set of values is known in advance. A raw request path has as many values as there are strings, and it grows fastest when the site is under attack. Every scan for The last row is the subtle one, because the number of series alive at any moment stays small. What grows without bound is the index, which has to remember every series that has ever existed within the retention window. A deploy every hour with a fresh container id produces a monitoring system that gets slower every week while the dashboard shows nothing unusual.<\/p>\n Memory is the binding constraint long before disk is. The head block holds every active series and its index in RAM, on the order of a few kilobytes per series, and a query touching many series needs room on top of that.<\/p>\n The failure is not gradual. Prometheus grows until the kernel kills it, restarts, replays the write-ahead log, grows again and gets killed again – usually within minutes of the first larger dashboard query. What looks like a broken exporter is a metric that gained a label three deploys ago.<\/p>\n The first query names the metric, the second names the label, and the third says whether the problem is a size or a leak. A head series count that rises steadily while nothing is being added is churn, and churn is always a label that changes when it should not.<\/p>\n These are worth running once on a healthy system, purely to know what normal looks like. A number without a baseline is not a diagnosis.<\/p>\n The cheapest fix happens at scrape time, before anything is stored. A Dropping is not the same as aggregating: the series for the individual paths disappear, and the counts they held are merged into whatever series remain. That is usually what was wanted anyway – almost nobody looks at a per-path request count, and the ones who do want it grouped by route rather than by URL.<\/p>\n Where the detail genuinely matters, the answer is to bound it at the source: label by Before a label is added, the question worth asking is what its value set is and who decides it. If the answer names a person or a configuration file, the label is bounded. If the answer is a visitor, a request, or an identifier, it is not – and it will be discovered later, from a Pi that no longer boots into a working Prometheus.<\/p>\n The same question also settles the privacy side of the matter, because the labels that are unbounded are exactly the ones that identify people. An email address in a metric label is a cardinality problem and a data protection problem in the same breath, and both are solved by not putting it there.<\/p>\nmethod<\/code> has five values and will still have five next year. status<\/code> has a dozen. instance<\/code> grows when a machine is added, which is a decision somebody makes.<\/p>\n\/wp-admin<\/code>, \/.env<\/code> and \/phpmyadmin<\/code> creates a permanent time series in a monitoring system that was watching a site which does not have those pages.<\/p>\n\n\n
\n \nLabel<\/th>\n Values<\/th>\n Bounded by<\/th>\n<\/tr>\n<\/thead>\n \n job<\/code>, instance<\/code><\/td>\na handful<\/td>\n the machines that exist<\/td>\n<\/tr>\n \n method<\/code>, status<\/code><\/td>\n5 to 12<\/td>\n the HTTP specification<\/td>\n<\/tr>\n \n handler<\/code>, route<\/code><\/td>\ntens<\/td>\n the routes in the code<\/td>\n<\/tr>\n \n path<\/code> (raw)<\/td>\nunbounded<\/td>\n nothing – it grows with scanners<\/td>\n<\/tr>\n \n user_id<\/code>, email<\/code><\/td>\nunbounded<\/td>\n nothing, and it belongs in no monitoring system<\/td>\n<\/tr>\n \n container_id<\/code>, pod<\/code><\/td>\nunbounded over time<\/td>\n nothing – a new value on every deploy<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n What It Costs in Practice<\/h2>\n
The Three Queries That Find It<\/h2>\n
# which metric names carry the most series\ntopk(10, count by (__name__)({__name__=~\".+\"}))\n\n# how many distinct values one label has, for one metric\ncount(count by (path) (http_requests_total))\n\n# total series in the head block, and how fast it is growing\nprometheus_tsdb_head_series\ndelta(prometheus_tsdb_head_series[1h])<\/code><\/pre>\nDropping a Label Is One Line<\/h2>\n
metric_relabel_configs<\/code> block drops the label from the incoming samples, and everything downstream shrinks by the factor that label contributed.<\/p>\nscrape_configs:\n - job_name: 'app'\n static_configs:\n - targets: ['app:9100']\n metric_relabel_configs:\n - regex: 'path|user_id|container_id'\n action: labeldrop<\/code><\/pre>\nhandler<\/code> or by a normalised route template rather than by the raw path, so that \/product\/12345<\/code> and \/product\/12346<\/code> become one series instead of two.<\/p>\nA Rule for New Metrics<\/h2>\n