Explore using Istio as a service mesh

[mjm]

On the surface, it would seem like Kubernetes already has pretty good support for deploying a mesh of distributed services: it will round robin requests between a set of pods behind one DNS name, and has ways to handle liveness and readiness. For what is currently deployed in MacStadium, that is good enough.

When we start looking to deploy more of our services on Kubernetes, though, Istio offers some features that we could find very useful. Ones that stand out to me:

• Automatic rate-limiting and circuit-breaking between our own services, to minimize how much destruction one buggy service can cause in the cluster
• Automatic secure TLS communication between services
• My personal favorite: ability to define routing rules for a service, which let you do staged rollouts! We could deploy new versions of a service and control how much traffic goes there
• All of the configuration is stored in Kubernetes resources, so it can be managed in Helm charts

[mjm]

This is even more compelling with this news: Google Integrates Istio Service Mesh into Kubernetes Service

I think this will be a big deal when we move services from Heroku onto Kubernetes. We should try to explore it in some less critical places first.