How to connect to a GKE private endpoint using the Identity Aware Proxy

Did you ever create a GKE cluster with a private endpoint? And found yourself in a world of pain trying to get your CI/CD pipelines to work? In this blog I will show you how to connect to a GKE private endpoint through the Identity Aware Proxy.

GKE CIS security control 6.6.4 require clusters with public access disabled. But, with public access disabled, your CI/CD system can no longer deploy to the cluster.

The most common ways to solve that is through the use private runners in the VPC or tunneling via a bastion host. But these are pretty big changes to the setup. Would it not be nice, if GKE master endpoints are directly accessible via IAP? The following diagram shows a way to do this:

In the diagram you see the following components:

• GKE client proxy is a true HTTPS proxy. It will only intercept proxy connection requests for the GKE cluster endpoints in your project. It will add an ID token as proxy-authorization header to the Identity Aware Proxy.
• GKE server proxy only forwards requests to the GKE cluster endpoint in your project.
• kubectl use the GKE client proxy, by pointing the proxy-url to the GKE client proxy at https://localhost:8080 and accepting the proxy certificate for https connections.

That is all there is to it. It is mostly transparent. Note that the IAP does not support websockets. This is why commands like kubectl exec will not work.

Installation

If you want to create a installation to play with, please follow the instructions at github.com/binxio/simple-iap-proxy. Although this setup is tailored for GKE, it is easy to extend the software to support generic proxy-ing into the cluster via IAP.

Conclusion

Thanks to the power of the ecosystem of golang, it was relatively straightforward to create setup to connect to private GKE master endpoints via the Identity Aware Proxy. I do hope that in the not-to-distant future, Google will provide a standard solution for this.

Disclaimer

Although we did test this setup on a moderately sized GKE configuration, it should be considered as beta software. Any feedback is much appreciated.

If you want to connect to a private GKE cluster from Google Cloud Build read this blog.
Image by PublicDomainPictures from Pixabay