Prometheus endpoint unprotected installs could expose sensitive data

Experts discovered several unprotected installs of open source event monitoring solution Prometheus that may expose sensitive data.

JFrog researchers have discovered multiple unprotected instances of open source event monitoring solution Prometheus that may leak sensitive data.

The solution scrapes real-time metrics from multiple endpoints, it is used by several major organizations such as Uber.

Prometheus’ retrieval job, also called the scraper, pulls data from target services, aggregates it, and passes it to the database.

JFrog researchers discovered numerous Prometheus endpoints exposed online that leak metric and label data, they were able to perform “a large-scale unauthenticated scraping of publicly available and non-secured” installs.

In January, the Transport Layer Security (TLS) and basic authentication support was introduced with the release of version 2.24.0,

Unfortunately, many Prometheus installs haven’t yet enabled these security features and JFrog researchers have focused their analysis on them. JFrog performed “a large-scale unauthenticated scraping of publicly available and non-secured Prometheus endpoints.

JFrog found nearly 27,000 unsecured installs using the Shodan search engine, and 43,000 hosts using ZoomEye.

“Using search engines like Shodan or ZoomEye it’s extremely easy to find tens of thousands of Prometheus endpoints. The most effective single query we’ve seen in Shodan, was to look for Prometheus endpoints by the Web UI’s favicon .” reads the post published by the experts. “This specific query (http.favicon.hash:-1399433489) returns almost 27K hosts in Shodan and 43K hosts in ZoomEye. By iterating automatically over these exposed endpoints, we’ve seen that 100% of the endpoints returned from this query had publicly-accessible data (meaning no authentication mechanisms were in place).”

Exposed data could be sensitive and could be used by threat actors to carry out further attacks against the organizations. Exposed data

Some of the exposed data login credentials in URL strings related to multiple services, infrastructure services, machine addresses and metadata labels, SSH public keys, environment variables for Kubelet, and more.

JFrog experts also warned of further security risks,associated with an optional management API that can be enabled via command line flags and that can be abused to delete all the saved metrics and to shut down the monitoring server.

“In our unauthenticated scraping effort, we discovered that ~15% of the exposed Prometheus endpoints had enabled API management, and ~4% had enabled database management. This means that right off the bat, an unauthenticated attacker can trivially shutdown and/or delete the metrics of these Prometheus endpoints. While our investigation clearly indicates this capability, to avoid harm or damage to users of those endpoints, we did not make any attempt to cause such a shutdown or a deletion as part of this research.” continues the post.

Before JFrog published the report I was alerted about the exposed install by the security researchers Anis Haboubi.

Researchers recommend using authentication and encryption mechanisms when deploying Prometheus to prevent the leak of sensitive information.

Follow me on Twitter: @securityaffairs and Facebook

[adrotate banner=”9″]

[adrotate banner=”12″]

Pierluigi Paganini

(SecurityAffairs – hacking, Prometheus)

[adrotate banner=”5″]

[adrotate banner=”13″]