Pierluigi Paganini June 11, 2020

Microsoft’s Azure Security Center (ASC) is warning of a hacking campaign that targets Kubeflow, a machine learning toolkit for Kubernetes.

Hackers are targeting Kubeflow servers with administration panel exposed online, Microsoft warns.

The tech giant has released a report today detailing a novel series of attacks against Kubeflow, a toolkit for deploying machine learning (ML) workflows on Kubernetes clusters.

The attacks have begun in April, threat actors aim at delivering a cryptocurrency miner on Kubernetes clusters running Kubeflow instances that were exposed to the internet.

“we’ll reveal a new campaign that was observed recently by ASC that targets Kubeflow, a machine learning toolkit for Kubernetes. We observed that this attack effected on tens of Kubernetes clusters.” reads the report published by Microsoft.

“Kubeflow is an open-source project, started as a project for running TensorFlow jobs on Kubernetes.”

Hackers are targeting Kubernetes clusters because nodes used from ML operations have huge computational capabilies making them ideal for fraudulent mining purposes.

But while the number of hijacked clusters is small in comparison to previous Kubernetes attacks, the profits for crooks and the financial losses to server owners are most likely much higher than other attacks seen before.

“Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs. This fact makes Kubernetes clusters that are used for ML tasks a perfect target for crypto mining campaigns, which was the aim of this attack.” continues the report.

In April, experts from Microsoft’s Azure Security Center observed the deployment of a suspect image, ddsfdfsaadfs/dfsdf:99, from a public repository on many different clusters. The analysis of the image revealed the presence of an XMRIG miner.

Kubeflow framework consists of many different services, which could include frameworks for training models, Katib and Jupyter notebook server, and more.

“Kubeflow is a containerized service: the various tasks run as containers in the cluster. Therefore, if attackers somehow get access to Kubeflow, they have multiple ways to run their malicious image in the cluster.” reads the report.

By default, the Kubeflow management panel is not exposed on the Internet and is only accessible from inside the Kubernetes cluster, but clusters targeted by the hackers were set up to expose the admin panel online.

Threat actors behind this campaign initially scan the Internet for exposed admin panel, then abused them to deploy new server images to Kubeflow clusters that were running XMRig.

“The attacker used an exposed dashboard (Kubeflow dashboard in this case) for gaining initial access to the cluster. The execution and persistence in the cluster were performed by a container that was deployed in the cluster.” continues the expert. “The attacker managed to move laterally and deploy the container using the mounted service account. Finally, the attacker impacted the cluster by running a cryptocurrency miner.”

To check if a cluster has been affected Microsoft recommends to the following procedure:

• Verify that the malicious container is not deployed in the cluster using the following command:

kubectl get pods –all-namespaces -o jsonpath=”{.items[*].spec.containers[*].image}”  | grep -i ddsfdfsaadfs

• In case Kubeflow is deployed in the cluster, make sure that its dashboard isn’t exposed to the internet: check the type of the Istio ingress service by the following command and make sure that it is not a load balancer with a public IP:

kubectl get service istio-ingressgateway -n istio-system

“However, this is the first time that we have identified an attack that targets Kubeflow environments specifically.” concludes Microsoft.

Pierluigi Paganini

(SecurityAffairs – Kubeflow, hacking)

[adrotate banner=”5″]

[adrotate banner=”13″]