Bare-metal ARM single-board computers such as the Raspberry Pi 4 and 5 operate under tight thermal and I\/O constraints. Sustained container workloads, frequent database write cycles, and intense network throughput frequently trigger silent hardware degradation. High SoC temperatures induce dynamic thermal throttling via the Broadcom firmware, reducing ARM clock frequencies without logging explicit errors to user application spaces. Similarly, continuous random write operations on MicroSD cards or USB-connected SSDs lead to flash block exhaustion, read\/write I\/O bottlenecks, and eventual kernel filesystem corruption.<\/p>\n
Implementing a Bare-Metal Performance Diagnostics Pipeline<\/strong> utilizing Prometheus<\/strong> and Grafana<\/strong> transforms opaque hardware states into observable, time-series metrics. By combining the standard node_exporter<\/code> daemon with a specialized custom textfile collector script, critical Raspberry Pi bare-metal indicators can be scraped every 15 seconds, evaluated for threshold violations, and visualized in real-time dashboards. Those indicators include SoC core temperatures, under-voltage flags, clock frequencies, RAM saturation, and storage I\/O wait times; the values supplied by vcgencmd<\/code> are refreshed once per minute by the scheduled collector script.<\/p>\n\nData flow of the diagnostics pipeline: vcgencmd<\/code> values reach Grafana via the textfile collector, Node Exporter and Prometheus \u2014 and trigger an alert above 75 \u00b0C or when the throttling bitmask is non-zero.<\/figcaption><\/figure>\n
Step-by-Step Implementation Guide<\/h2>\n
Step 1: Deploying Node Exporter and the Raspberry Pi Textfile Collector<\/h3>\n
To capture standard Linux kernel telemetry alongside proprietary Broadcom hardware statistics, install node_exporter<\/code> and create a dedicated directory for custom metric textfiles:<\/p>\n
Create a custom bash script at \/opt\/scripts\/rpi-hardware-metrics.sh<\/code> to extract SoC temperature, core voltage, and hardware throttling flags using vcgencmd<\/code>:<\/p>\n
#!\/usr\/bin\/env bash\n# \/opt\/scripts\/rpi-hardware-metrics.sh\n# Extracts bare-metal SoC metrics for Prometheus textfile collector\n\nTEXTFILE_DIR=\"\/var\/lib\/node_exporter\/textfile_collector\"\nTMP_FILE=\"$TEXTFILE_DIR\/rpi_metrics.prom.$$\"\nFINAL_FILE=\"$TEXTFILE_DIR\/rpi_metrics.prom\"\n\n# 1. Extract CPU Core Temperature (degrees Celsius)\nTEMP_C=$(vcgencmd measure_temp | grep -oE '[0-9]+([.][0-9]+)?')\necho \"# HELP rpi_soc_temperature_celsius Broadcom SoC Core Temperature\" > \"$TMP_FILE\"\necho \"# TYPE rpi_soc_temperature_celsius gauge\" >> \"$TMP_FILE\"\necho \"rpi_soc_temperature_celsius $TEMP_C\" >> \"$TMP_FILE\"\n\n# 2. Extract SoC Core Voltage (Volts)\nVOLT_V=$(vcgencmd measure_volts core | grep -oE '[0-9]+([.][0-9]+)?')\necho \"# HELP rpi_soc_core_voltage_volts Broadcom SoC Core Voltage\" >> \"$TMP_FILE\"\necho \"# TYPE rpi_soc_core_voltage_volts gauge\" >> \"$TMP_FILE\"\necho \"rpi_soc_core_voltage_volts $VOLT_V\" >> \"$TMP_FILE\"\n\n# 3. Extract Throttling Hex Code as integer bitmask\nTHROTTLE_HEX=$(vcgencmd get_throttled | cut -d'=' -f2)\nTHROTTLE_DEC=$((THROTTLE_HEX))\necho \"# HELP rpi_soc_throttled_bitmask Bitmask representing under-voltage or thermal throttling flags\" >> \"$TMP_FILE\"\necho \"# TYPE rpi_soc_throttled_bitmask gauge\" >> \"$TMP_FILE\"\necho \"rpi_soc_throttled_bitmask $THROTTLE_DEC\" >> \"$TMP_FILE\"\n\nmv \"$TMP_FILE\" \"$FINAL_FILE\"<\/code><\/pre>\n
Make the script executable and schedule execution every minute via crontab:<\/p>\n
Restart the service to apply the flags: systemctl restart prometheus-node-exporter<\/code>.<\/p>\n
Step 3: Configuring the Prometheus Time-Series Scraper<\/h3>\n
Deploy Prometheus using Docker Compose with network_mode: host<\/code> and configure a 15-second scraping interval pointing to the local node exporter target:<\/p>\n
Step 4: Deploying Grafana and Importing Bare-Metal Panels<\/h3>\n
Launch Grafana via Docker Compose, also with network_mode: host<\/code>, connect the Prometheus database (http:\/\/127.0.0.1:9090<\/code>) as the default data source, and create dashboard panels utilizing the following PromQL queries:<\/p>\n
\n
SoC Core Temperature Panel:<\/strong> rpi_soc_temperature_celsius<\/code> (Alert threshold configured at > 75\u00b0C).<\/li>\n
Under-Voltage & Throttling Alert:<\/strong> rpi_soc_throttled_bitmask > 0<\/code> (Indicates inadequate power supplies or severe overheating, now or at any point since the last boot; stays active until the next reboot).<\/li>\n
Step 5: Quality Assurance and Stress Testing<\/h3>\n
To confirm that the telemetry pipeline accurately captures hardware exhaustion, stress-testing validation must be performed:<\/p>\n
\n
Synthetic I\/O Stress Validation:<\/strong> Execute an intensive disk benchmark using fio<\/code> (e.g., fio --name=test --ioengine=libaio --rw=randwrite --bs=4k --size=1G<\/code>) and verify that the Grafana I\/O Wait panel shows real-time storage latency spikes.<\/li>\n
Thermal Throttling Simulation:<\/strong> Stress all ARM CPU cores via stress-ng --cpu 4 --timeout 120s<\/code> while monitoring rpi_soc_temperature_celsius<\/code> in Prometheus to confirm accurate temperature tracking up to cooling fan activation thresholds.<\/li>\n
Textfile Metric Parsing Audit:<\/strong> Execute curl -s http:\/\/127.0.0.1:9100\/metrics | grep rpi_<\/code> in the terminal to verify that custom voltage and throttling metrics are cleanly exposed without formatting syntax errors.<\/li>\n<\/ol>\n
Summary and Measurable Added Value<\/h2>\n
What is achieved:<\/strong> Deployment of an autonomous bare-metal hardware diagnostic pipeline on Raspberry Pi hardware using Prometheus time-series scraping, custom SoC textfile collectors, and Grafana monitoring dashboards.<\/p>\n
Resulting added value:<\/strong><\/p>\n
\n
Early Detection of Storage Failure:<\/strong> Continuous monitoring of disk I\/O wait times and flash wear allows preventative SD card or SSD replacement before catastrophic filesystem corruption occurs.<\/li>\n
Prevention of Thermal Throttling:<\/strong> Real-time visibility into SoC temperatures and throttling bitmasks guarantees that containerized services run at maximum clock frequency without silent performance degradation.<\/li>\n
Power Supply Quality Control:<\/strong> Immediate logging of under-voltage flags (bit 0, bitmask 0x1<\/code>) identifies failing USB-C power adapters or excessive peripheral power draw before hardware crashes occur.<\/li>\n<\/ul>\n
\n
Questions and answers<\/h2>\n
Why does the throttling alert stay active until the next reboot after a single incident?<\/h3>\n
Because get_throttled returns two kinds of bits. The lower bits describe the current state, among them bit 0 for under-voltage and bit 2 for throttling. The bits from 16 upward record that the same thing has happened at some point since boot, and they stay set until the Raspberry Pi restarts. A brief voltage dip in the morning therefore yields something like 0x50000 (under-voltage and throttling have occurred), 327680 in decimal, and this value stays above zero for the rest of the day.<\/p>\n
For evaluation it pays to separate the two. PromQL supports the % operator, and rpi_soc_throttled_bitmask % 16 > 0<\/code> checks only the four lower bits, meaning the current state; rpi_soc_throttled_bitmask >= 65536<\/code> reports that something has happened since boot. Clearer still is to have the script write a separate metric for each bit.<\/p>\n
Then there is the script’s timing: it runs once a minute, and get_throttled only reports the state at that moment. Throttling lasting a few seconds between two runs does not show up in the lower bits, but it does in the bits from 16 upward. For brief under-voltage in particular, the recorded part is therefore the more reliable one.<\/p>\n
Can Prometheus in a container reach the target 127.0.0.1:9100?<\/h3>\n
Only with network_mode: host. In a regular Docker network, 127.0.0.1 is the container itself, while node_exporter runs outside it directly on the host, so the target stays down permanently. The same applies to Grafana: if both containers share a Docker network, the data source address uses the name of the Prometheus service instead of 127.0.0.1.<\/p>\n<\/div>\n
![\"Data](\"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/diagrams\/rpi-prometheus-grafana-pipeline-en.png\")