Uber Automates Dependency Analysis Across Its Service Mesh

Yes, Uber has automated dependency analysis across its service mesh. The company built internal tooling that continuously maps service-to-service dependencies, detects anomalies, and feeds that data into load testing, capacity planning, and incident response. This article explains how that automation works, why Uber needed it, and what other engineering teams can learn from the approach.

Background: Why Uber Needed Dependency Analysis

Uber runs thousands of microservices in a distributed architecture. Each ride, delivery, or payment event touches many services: the mobile app calls an API gateway, which calls trip dispatch, which calls pricing, which calls payment, and so on. When one service fails or slows down, the impact can cascade through the entire mesh. Without an accurate map of these dependencies, engineers cannot predict which services will be affected by a change, how much traffic a service can handle, or where to focus debugging during an outage.

Manual dependency mapping—drawing diagrams or maintaining spreadsheets—does not scale at Uber’s size. Services are added, removed, and repointed constantly. A static map becomes outdated within days. Uber needed a system that could discover dependencies automatically, update them in near real time, and make the information actionable for both humans and automated systems.

How Uber’s Dependency Analysis Works

Uber’s approach combines several techniques to build a live dependency graph:

• Distributed tracing: Uber uses its own tracing system (based on the open-source Jaeger project) that records every request as it passes through services. Each span includes the service name, operation, and parent-child relationships. Aggregating these spans over a time window reveals which services call which others, and how often.
• Service mesh telemetry: Uber’s service mesh (built on Envoy) captures metadata about every RPC between services—source, destination, latency, error rate, and protocol. This data is streamed to a central store and processed to build the dependency graph.
• Graph construction: The telemetry is aggregated into a graph where nodes are services and edges are calls. Edge weights represent traffic volume or request count. The graph is updated continuously, typically with a delay of a few minutes, so it reflects current behavior rather than a stale snapshot.
• Anomaly detection: The system compares the current graph to historical baselines. New edges, removed edges, or sudden changes in traffic volume trigger alerts. For example, if a service starts calling a new database or an external API, the system flags it so engineers can verify it is intentional.

This automated pipeline replaces manual discovery. Engineers can query the graph to answer questions like “What services depend on this payment service?” or “What does this service call when handling a ride request?”

Automation in Practice: Load Testing and Capacity Planning

One of the primary uses of the dependency graph is load testing. Uber runs large-scale load tests against its production-like environment to see how services behave under peak traffic. The dependency graph tells the testing system which services to include in the test and how to simulate realistic call patterns. Without an accurate graph, a load test might miss a critical dependency, leading to false confidence.

Capacity planning also relies on the graph. When Uber expects a surge in demand (e.g., New Year’s Eve), the capacity team uses the dependency graph to identify which services will be stressed and how much capacity each needs. The graph’s traffic volumes help estimate load, and the anomaly detection helps spot services that might become bottlenecks because they are newly introduced or suddenly receive more traffic.

Common Problems and Misconceptions

A common misconception is that dependency analysis is the same as service discovery. Service discovery (e.g., using Consul or etcd) tells you which instances of a service exist, but it does not tell you which services call each other. Dependency analysis is about call relationships, not just instance availability.

Another problem is that dependency graphs can be noisy. Not every call is equally important; some are rare, some are asynchronous, and some are retries. Uber’s system filters and aggregates to focus on meaningful edges—for example, ignoring calls that occur less than a certain number of times per minute. Without this filtering, the graph would be overwhelming and less useful.

Handling asynchronous communication is also tricky. A service might send a message to a queue, and a worker processes it later. The dependency is not visible in a simple request/response trace. Uber’s tracing system correlates asynchronous operations using propagation headers, so the graph includes these indirect dependencies as well.

Comparison with Open-Source Tools

Uber’s solution is custom-built, but it shares concepts with open-source tools. Jaeger, which Uber originally developed, provides tracing data that can be used to infer dependencies. Zipkin and Apache SkyWalking do similar things. Service mesh tools like Istio and Linkerd produce telemetry that can be fed into a graph database like Neo4j to visualize dependencies.

The difference is scale and integration. Uber’s system is tightly coupled with its internal infrastructure, including its deployment system, alerting, and incident management. Open-source tools are more generic and require custom glue code to achieve the same level of automation. For a team starting out, using an existing tracing system and building a simple graph aggregator is a reasonable first step.

Costs and Trade-offs

Automated dependency analysis is not free. It requires significant infrastructure to collect, store, and process telemetry. Uber’s scale means it processes trillions of spans and RPC records per day, which demands substantial storage and compute resources. The system also adds latency overhead to requests, though Uber keeps it low (typically under a few percent) by sampling and using efficient encoding.

There is also a human cost: the tooling needs to be maintained and evolved as the architecture changes. However, the benefit—accurate, up-to-date dependency information—usually outweighs the cost, because it reduces outage time and improves planning accuracy.

Alternatives and Future Directions

Teams that do not have Uber’s resources can use simpler alternatives: manual diagrams for small systems, or open-source tracing tools with basic dependency views. Some teams use static analysis of code to infer dependencies, but that misses runtime dynamics like service discovery and dynamic routing.

Looking forward, Uber is likely integrating dependency analysis with AI-based anomaly detection to predict failures before they happen. The dependency graph can be used to simulate failure scenarios (e.g., “what if this service goes down?”) and to guide chaos engineering experiments.

Frequently Asked Questions

Does Uber use a standard service mesh like Istio?

Uber built its own service mesh on top of Envoy, not using Istio directly. This gives them more control and optimization for their scale. The dependency analysis is built on top of that mesh’s telemetry.

How does Uber handle dependencies that change frequently?

The system updates the graph continuously, typically within minutes. It also uses anomaly detection to alert engineers when new dependencies appear or existing ones disappear, so changes are reviewed rather than silently accepted.

Can dependency analysis be used for security?

Yes, the dependency graph can reveal unexpected connections that might indicate a security issue, such as a service calling an external endpoint it should not. Uber uses the graph for security reviews and to enforce network policies.

Is the dependency graph available to all engineers?

Yes, Uber provides a UI and API for engineers to query the graph. It is integrated into their developer tools, so engineers can see dependencies when they are investigating an incident or planning a change.

What is the biggest challenge in automating dependency analysis?

The biggest challenge is data volume and accuracy. At Uber’s scale, collecting and processing telemetry without losing fidelity requires careful engineering. Also, distinguishing meaningful dependencies from noise is difficult, but essential for the graph to be useful.

Related

• Scaling Telecom Manufacturing: Assembly to R&D
• Print in Place 3D Printer Model Meaning
• Remove GPT Protective Partition? Safe Steps