API gateway vs. service mesh vs. message queue - when do I use what?

Note: This post was first published in March 2020 and has since been thoroughly revised to incorporate recent developments such as the Kubernetes Gateway API and eBPF-based service meshes.obtain.

Microservices solve one problem, but they also create a new one: Once the monolithic application has been broken down into many small services, it must be determined how these services communicate with each other and with the outside world. Anyone making this decision for their cluster will almost inevitably end up with three components: an API gateway, a service mesh, and a message queue. In most cases, however, it’s not an either-or decision. Many DevOps teams use a combination of these approaches—for example, an API gateway for incoming traffic from outside, a service mesh for communication between services, and a message queue for asynchronous tasks. This article categorizes the three concepts, highlights their overlaps, and provides guidance on when to use each component.

What is actually the problem?

Here is a brief analysis of the problem: For microservices to function as a distributed system, they must overcome a long list of challenges.

Elasticity
There can be dozens or even hundreds of instances of a particular microservice, any of which can fail at any given time—for example, due to a node restart, an out-of-memory error, or a failed deployment. A system that doesn’t account for this will bring down the entire request with every single failure. Elasticity means: The failed payment service pod simply disappears from the list of responsive instances, and a new one takes over—without the calling application even noticing.

Load balancing and automatic scaling
With potentially hundreds of endpoints capable of handling a request, routing and scaling are anything but trivial. It’s not enough to simply distribute requests on a round-robin basis: An instance that is currently processing a computationally intensive request should receive new requests less frequently than one that is currently idle. More precise routing and scaling decisions—such as those based on current load rather than a fixed distribution—are among the most effective ways to control costs in large-scale architectures.

Service detection
The more distributed an application is, the harder it becomes to determine which instances of a service currently exist and are accessible—pod IP addresses change with every restart. Service discovery solves this by having services register under a stable name (e.g.,. payment service) rather than under a fixed IP address, and a directory that continuously resolves this name to the currently active instances.

Tracking and monitoring
In a microservices architecture, a single user request can pass through multiple services—for example, a purchase process might go through the auth service, payment service, inventory service, and notification service in succession. If something goes wrong anywhere in this chain, it is nearly impossible to determine which of the four services caused the problem without end-to-end tracing. Distributed tracing links all the steps of a request via a common trace ID, allowing the entire path to be reconstructed afterward.

Versioning
As systems grow, APIs must be further developed without breaking existing consumers—another team that's still working against /api/v1/orders work must not suddenly come to nothing just because, internally, it has long since /api/v2/orders is in use. As a result, multiple API versions often need to be offered in parallel and phased out with a clear end date (deprecation), rather than rolling out breaking changes without warning.

The solutions

In this article, we present the three key approaches to solving these problems: service meshes, API gateways (including their Kubernetes-native variant, the Gateway API), and message queues. Of course, there are also a number of other approaches that could be used, ranging from simple static load balancing via fixed IP addresses to central orchestration servers. In this post, however, we’ll focus on the most popular and, in many respects, the most sophisticated options.

API Gateways

An API gateway is the big brother of the good old reverse proxy for HTTP calls. It is a scalable server, normally connected to the Internet, that can receive requests from both the public Internet and internal services and route them to the most appropriate microservice instance. API gateways provide a number of helpful features, including load balancing and integrity checks, API versioning and routing, request validation and authorization, data transformation, analysis, logging, SSL termination, and more. Examples of popular open source API gateways include. Kong or Tyk. Most cloud providers also offer their own implementation, such as AWS API Gateway, Azure API Management, or Google Cloud Endpoints.

The advantages
API gateways offer powerful features, are comparatively low in complexity, and are easy for seasoned web veterans to understand. They provide solid protection against the public Internet and perform many repetitive tasks such as user authentication or data validation.

The disadvantages
API gateways are fairly centralized. They can be deployed in a horizontally scalable manner. However, unlike service meshes, new APIs have to be registered or the configuration changed at a central location. From an organizational perspective, they should therefore also only be managed by one team.

Kubernetes-native variant: Gateway API
Running microservices on Kubernetes is now virtually impossible without using another feature: the Gateway API. It is the official successor to the classic Kubernetes Ingress object and addresses its biggest weakness: the reliance on controller-specific annotations. These annotations are not portable between different Ingress controllers. The core resources of the Gateway API—GatewayClass, Gateway, HTTPRoute, GRPCRoute, TLSRoute, and ReferenceGrant—have reached „General Availability” (GA) status and are considered production-ready. Of particular note: The Ingress NGINX controller, which is used in many clusters, has been officially retired since March 2026. Several highly critical CVEs (including one with a CVSS score of 8.8) have already been published for the abandoned codebase, for which no further patches will be released. Anyone still running on classic Ingress should therefore plan to migrate to the Gateway API to avoid continuing to operate without security updates.

Service meshes

Service meshes are decentralized, self-organizing networks between microservice instances that handle load balancing, endpoint discovery, integrity checks, monitoring, and tracing. They operate by adding a small agent to each instance, called a "sidecar." The service mesh mediates traffic and registration of instances, handles metrics collection and maintenance. While most service meshes are conceptually decentralized, they have one or more centralized elements to collect data or provide admin interfaces. Popular service mesh examples include Istio, Linkerd, or Hashicorp's Consul.

Advantages
Service meshes are more dynamic and can easily change shape to accommodate new features and endpoints. Their decentralized nature makes it easier to work on microservices in isolated teams.

Disadvantages
Service meshes are based on many moving parts and can therefore become very complex very quickly. For example, fully leveraging Istio requires deploying a separate traffic manager, telemetry collector, certificate manager, and sidecar process for each node. They are also a relatively recent development for something that should form the backbone of your IT architecture.

The Next Step: Sidecar-Free Meshes
The main criticism of the classic sidecar model—an additional proxy process per pod that consumes resources and increases latency (see comparison table)—is now addressed by eBPF-based approaches. eBPF (extended Berkeley Packet Filter) allows network logic to be executed directly in the Linux kernel instead of running it in a separate sidecar process per pod. Cilium Service Mesh uses this for a sidecar-less approach at the node level. Istio itself also offers, with Ambient Mesh now supports an operating mode that does not require a sidecar per pod. For new clusters, the following approach is therefore increasingly recommended: Use Cilium as the CNI (Container Network Interface), test its native mesh capabilities with Hubble for observability—and only add a separate mesh layer like Istio if additional needs arise, rather than assuming it’s required from the outset.

Message queues

At first glance, comparing service meshes to a message queue seems like comparing apples to oranges: They are completely different things, but they solve the same problem, albeit in very different ways.

With a message queue, you can establish complex communication patterns between services by decoupling senders and receivers. You accomplish this using a number of measures, such as topic-based routing or publish-subscribe messaging, as well as buffered processing, which makes it easier for multiple instances to handle different aspects of a task over time.

Message queues have been around for ages, resulting in a wide range of options: Popular open-source alternatives include Apache Kafka, AMQP brokers such as RabbitMQ, and Apache ActiveMQ Artemis. However, they are also provided by the respective cloud providers.

Advantages
Simply decoupling senders and receivers is an effective concept that eliminates the need for a number of other concepts such as integrity checks, routing, endpoint detection, or load balancing. Instances can select relevant tasks from a buffered queue as soon as they are ready. This is particularly effective when automatic orchestration and scaling decisions are based on the number of messages in each queue, resulting in resource-efficient systems.

Disadvantages
Message queues are not good at request / response communication. Some allow you to match this to existing concepts, but it is not really what they are made for. Because of their buffering, they can also add significant latency to a system. They are also quite centralized (although horizontally scalable) and can be quite expensive on a large scale.

Comparison Table

With classic sidecar mesh (Istio with Envoy sidecar), the following occur: two additional hops per call (sending a sidecar + receiving a sidecar), not just one. With Ambient Mesh, this is partially eliminated due to the split node-proxy architecture.

So, when should you choose which solution?
Actually, this is not necessarily an either-or decision. In fact, it may make perfect sense to provide the publicly available API with an API gateway, run a service mesh for inter-service communication, and support things with a message queue for asynchronous task scheduling. A service mesh can work with an API gateway to efficiently accept external traffic and then effectively forward that traffic once it is on the network. The combination of these technologies can be a powerful way to ensure application availability and resiliency while ensuring that your applications can be used with ease.

In a deployment with an API gateway and a service mesh, incoming traffic from outside the cluster is routed first through the API gateway and then into the mesh. The API gateway can handle authentication, edge routing, and other edge functions, while the service mesh provides detailed observation and control of your architecture.

However, if you want to focus only on communication between services, one possible answer could be:

• If you are already running an API gateway for your public-facing API, you can keep the complexity just as low and reuse it for communication between services.
• If you work in a large organization with isolated teams and poor communication, a service mesh gives you maximum independence so you can easily add new services over time.
• If you are designing a system where individual steps are spread out over time, such as a YouTube-like service where uploading, processing, and publishing videos can take a few minutes, use a message or task queue to do this.

What the future holds

The 2020 forecast in this article has since come true: API gateways and service meshes have converged more closely than was foreseeable at the time—the Kubernetes Gateway API now handles routing tasks that used to be the exclusive domain of the API gateway, and modern service meshes like Istio Ambient Mesh or Cilium are increasingly moving away from the resource-intensive sidecar model. The trend here is not toward „more and more mesh,“ but rather toward targeted use: Teams are finding that a modern CNI like Cilium already covers much of what used to require a full-fledged service mesh—the added complexity isn’t worth it until the system reaches a certain size. This consolidation is expected to continue in the coming years: fewer separate components, more functionality directly in the kernel (eBPF) and within the Kubernetes platform itself.

Anyone who deploys a service mesh, API gateway, or message queue also operates the underlying infrastructure. ScaleUp offers SCS-Certified Managed Kubernetes and Cloud Hosting to 100 % in our own German data centers – Open source, based on OpenStack and Kubernetes.

Sources:

https://arcentry.com/blog/api-gateway-vs-service-mesh-vs-message-queue/
https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/
https://gateway-api.sigs.k8s.io/