# Apache Dubbo Administrator & Architecture Guide

This guide is for the people who **operate, integrate, extend, and reason about** Apache Dubbo: platform engineers, SREs, architects, and contributors. Where the [User Guide](apache__dubbo__UserGuide.md) shows how to *build with* Dubbo, this guide explains how Dubbo is *built* — its layered architecture, the SPI extension model that makes every component swappable, the scope-model domain that governs lifecycle, the deployment topology, the security surface, and the operational levers you pull in production.

It is intentionally exhaustive. The goal is that an engineer (or an AI) could understand, operate, harden, and contribute to a Dubbo-based system from this document alone.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Architecture Overview](#2-architecture-overview)
3. [The SPI Extension Mechanism](#3-the-spi-extension-mechanism)
4. [The Scope-Model Domain](#4-the-scope-model-domain)
5. [Repository & Module Map](#5-repository--module-map)
6. [The Invocation Pipeline](#6-the-invocation-pipeline)
7. [Service Discovery Internals](#7-service-discovery-internals)
8. [Requirements & Dependencies](#8-requirements--dependencies)
9. [Deployment](#9-deployment)
10. [Security](#10-security)
11. [Operations](#11-operations)
12. [Observability](#12-observability)
13. [Performance Tuning](#13-performance-tuning)
14. [Known Issues & Tech Debt](#14-known-issues--tech-debt)
15. [Contributing](#15-contributing)
16. [Quick Reference](#16-quick-reference)

---

## 1. Executive Summary

Apache Dubbo is a **Java microservices framework** delivered as a large Maven multi-module project (roughly 4,800 files across ~150 modules). It is not an application you run; it is a library set your applications depend on. Its defining architectural property is **pervasive extensibility**: almost every behavior — protocol, transport, serialization, registry, load balancer, cluster strategy, router, filter, proxy factory — is a named **SPI extension** resolved at runtime from the classpath. This is what lets one codebase support Triple/HTTP-2, classic TCP, REST, Zookeeper, Nacos, multiple serializers, and a service mesh without conditionals.

| Aspect | Summary |
|---|---|
| **Language / build** | Java (source level 1.8, runs on JDK 8–25), Maven multi-module, `${revision}` = `3.3.7-SNAPSHOT` (this analysis); current stable line `3.3.x` |
| **Core idea** | Interface-based RPC + automatic service discovery + traffic management, all behind swappable SPI extensions |
| **Domain model** | Three-tier scope hierarchy: `FrameworkModel` → `ApplicationModel` → `ModuleModel` |
| **Config carrier** | The `URL` object — every component is parameterized by a URL |
| **Default stack** | Protocol `dubbo` (TCP), serialization `hessian2`, cluster `failover`, loadbalance `random`, proxy `javassist` |
| **Discovery** | Application-level service discovery (Dubbo 3) with interface-level compatibility |
| **Transport** | Netty 4 (primary), plus HTTP/1.2/3 and WebSocket remoting modules |
| **Operations** | A built-in QOS port (telnet + HTTP) for runtime inspection and control |

> [!NOTE]
> This guide is generated from source at commit `ee9c190`. Because AST extraction does not parse Java, all reference data here (config fields, extension names, command names, SPI defaults) was read directly from the framework's source and SPI declaration files, not inferred.

---

## 2. Architecture Overview

Dubbo is organized as a stack of layers. A call enters at the top (your interface proxy) and descends through cluster, protocol, exchange, transport, and serialization before hitting the wire.

```
        ┌──────────────────────────────────────────────────────────┐
 config │ ServiceConfig / ReferenceConfig / @DubboService / Bootstrap│  ← you configure here
        ├──────────────────────────────────────────────────────────┤
  proxy │ ProxyFactory (javassist)  — interface ⇄ Invoker            │
        ├──────────────────────────────────────────────────────────┤
registry│ RegistryFactory / ServiceDiscovery / ServiceNameMapping    │  ← discovery + metadata
        ├──────────────────────────────────────────────────────────┤
 cluster│ Cluster (failover…) → Directory → Router → LoadBalance     │  ← fault tolerance + routing
        ├──────────────────────────────────────────────────────────┤
monitor │ Filter chain (validation, auth, metrics, tracing, cache…)  │  ← cross-cutting
        ├──────────────────────────────────────────────────────────┤
protocol│ Protocol (dubbo / tri / rest2 / grpc / injvm)              │  ← Invoker ⇄ Exporter
        ├──────────────────────────────────────────────────────────┤
exchange│ Exchanger / HeaderExchange — request/response semantics    │
        ├──────────────────────────────────────────────────────────┤
transport│ Transporter (Netty4) — Channel, Server, Client            │
        ├──────────────────────────────────────────────────────────┤
 serialize│ Serialization (hessian2 / fastjson2)                     │
        └──────────────────────────────────────────────────────────┘
```

Three properties tie the layers together:

1. **The `URL` is the contract.** Every layer reads the parameters it cares about from a shared `org.apache.dubbo.common.URL`. A registry URL like `registry://zk:2181/...?registry=zookeeper` is rewritten as it descends — `RegistryProtocol` turns it into a `dubbo://provider:20880/...` URL for the actual call. This uniform carrier is why the layers compose without bespoke glue.

2. **Every layer is an SPI.** The boundary between layers is a Java interface annotated `@SPI`, and the concrete implementation is selected by a name in the URL. See [§3](#3-the-spi-extension-mechanism).

3. **Wrappers add cross-cutting behavior.** When the SPI loader instantiates, say, a `Protocol`, it automatically wraps it in `ProtocolFilterWrapper`, `ProtocolListenerWrapper`, `ProtocolSerializationWrapper`, and `ProtocolSecurityWrapper`. This is how filters, listeners, and security are woven in without the core protocol knowing about them.

---

## 3. The SPI Extension Mechanism

This is the single most important thing to understand about Dubbo. Everything else follows from it.

### 3.1 `@SPI`, `ExtensionLoader`, and named extensions

An extension point is a Java interface annotated `@SPI`, located in `org.apache.dubbo.common.extension`. Implementations are declared in `META-INF/dubbo/internal/<fully.qualified.InterfaceName>` files as `name=implementationClass` lines — Dubbo's enhanced version of the JDK `ServiceLoader`.

```java
@SPI(value = "dubbo", scope = ExtensionScope.FRAMEWORK)
public interface Protocol { /* ... */ }
```

```text
# META-INF/dubbo/internal/org.apache.dubbo.rpc.Protocol
dubbo=org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol
tri=org.apache.dubbo.rpc.protocol.tri.TripleProtocol
grpc=org.apache.dubbo.rpc.protocol.tri.GrpcProtocol
rest2=org.apache.dubbo.rpc.protocol.tri.RestProtocol
injvm=org.apache.dubbo.rpc.protocol.injvm.InjvmProtocol
mock=org.apache.dubbo.rpc.support.MockProtocol
```

Code obtains an extension by name:

```java
Protocol dubbo = applicationModel.getExtensionLoader(Protocol.class).getExtension("dubbo");
```

The `@SPI` value (`"dubbo"`) is the **default** used when the URL doesn't name one.

### 3.2 Adaptive extensions

Most layers don't know which implementation to use until call time, when the protocol is in the URL. The `@Adaptive` mechanism generates a dispatcher that reads the relevant URL parameter and delegates to the right concrete extension:

```java
protocolSPI = getExtensionLoader(Protocol.class).getAdaptiveExtension();
// at call time, protocolSPI.refer(type, url) picks DubboProtocol or TripleProtocol
// based on url.getProtocol()
```

This is why `ReferenceConfig` holds a single `Protocol protocolSPI` yet transparently routes `dubbo://` to `DubboProtocol` and `tri://` to `TripleProtocol`.

### 3.3 Activate (conditional auto-wiring)

`@Activate` marks extensions (especially **filters** and **routers**) that should be auto-included when certain conditions hold (provider vs consumer side, a URL parameter present, etc.). The cluster and filter chains are assembled by collecting all `@Activate` extensions that match the current URL — which is how the validation filter, access-log filter, metrics filter, and tracing filter all insert themselves without explicit registration.

### 3.4 Extension scopes

`ExtensionScope` controls how widely an extension instance is shared:

| Scope | Shared across | Examples |
|---|---|---|
| `FRAMEWORK` | The whole JVM | `Protocol`, `ProxyFactory`, `Serialization` |
| `APPLICATION` | One application | `RegistryFactory` |
| `MODULE` | One module | `ConfigInitializer`, `ConfigPostProcessor` |

> [!IMPORTANT]
> When you extend Dubbo, you do **not** fork it. You implement the SPI interface, drop a `META-INF/dubbo/<InterfaceName>` file in your jar, and select your extension by name in config. This is the intended extension path for protocols, registries, serializers, filters, routers, and load balancers. The `dubbo-compiler` and `dubbo-native` modules support build-time stub generation and GraalVM native-image hints for this model.

---

## 4. The Scope-Model Domain

Dubbo 3 replaced global static state with an explicit, three-tier **scope-model** hierarchy rooted in `org.apache.dubbo.rpc.model`. This is what allows multiple isolated Dubbo "applications" in one JVM (multi-tenancy, test isolation, library embedding).

```
FrameworkModel              (one per JVM by default; shared framework services)
  └── ApplicationModel       (one per application; owns config, registries, deployers)
        └── ModuleModel       (one per module; owns a set of services/references)
```

| Model | Owns | Lifecycle driver |
|---|---|---|
| **`FrameworkModel`** | Framework-scoped extensions, the shared executor repository | `FrameworkModelCleaner` (destroy listener) |
| **`ApplicationModel`** | `ConfigManager` (all config beans), application deployer, service repository, bean factory | `DefaultApplicationDeployer` |
| **`ModuleModel`** | The module's exported services and references | `DefaultModuleDeployer` |

Each model carries a `ScopeBeanFactory` (a lightweight DI container) and a **Deployer** that drives startup/shutdown. `ConfigScopeModelInitializer` registers the deployers and the `DefaultConfigValidator` when the models are created.

> [!NOTE]
> The deployer is the orchestration heart. `DefaultApplicationDeployer.start()` initializes config, starts the config center and metadata report, exports the internal metadata service, registers services, and starts the QOS server. When Dubbo is embedded in Spring, modules are marked `isLifeCycleManagedExternally()` so Spring drives their lifecycle and Dubbo's shutdown hook waits for Spring before tearing down (see `DubboShutdownHook`).

### 4.1 Why this matters operationally

- **Graceful shutdown** is coordinated at the `ApplicationModel` level: the shutdown hook flips all services to read-only, drains in-flight calls up to `shutwait`, and only then destroys the application.
- **Multiple applications in one JVM** each get their own `ApplicationModel`, so their registries, config, and services don't collide — useful for gateways and test harnesses.
- **Config resolution** is per-`ApplicationModel` via its `ConfigManager`, which is why the precedence rules in the User Guide are evaluated per application.

---

## 5. Repository & Module Map

Dubbo's ~150 Maven modules group into functional families. The table lists the families and the headline modules in each; the suffix convention is `dubbo-<area>-<impl>`.

| Family | Key modules | Responsibility |
|---|---|---|
| **Core** | `dubbo-common` | URL, SPI/`ExtensionLoader`, scope models, config bean classes, utilities, constants |
| **Config** | `dubbo-config-api`, `dubbo-config-spring`, `dubbo-config-spring6` | `ServiceConfig`/`ReferenceConfig`, `DubboBootstrap`, Spring annotations & XML, autowiring |
| **RPC / protocols** | `dubbo-rpc-api`, `dubbo-rpc-dubbo`, `dubbo-rpc-triple`, `dubbo-rpc-injvm` | `Invoker`/`Exporter`/`Protocol`, the dubbo and Triple protocols, in-JVM calls |
| **Cluster** | `dubbo-cluster` | Cluster strategies, directories, routers (condition/tag/script/affinity/mesh), load balancers |
| **Registry** | `dubbo-registry-api`, `-zookeeper`, `-nacos`, `-multicast`, `-multiple` | Service registration & discovery, app-level discovery, migration |
| **Remoting** | `dubbo-remoting-api`, `-netty4`, `-netty`, `-http12`, `-http3`, `-websocket`, `-zookeeper-curator5` | Transports, channels, the exchange layer |
| **Serialization** | `dubbo-serialization-api`, `-hessian2`, `-fastjson2` | Wire serializers + the security wrapper |
| **Metadata** | `dubbo-metadata-api`, `-report-zookeeper`, `-report-nacos`, `-definition-protobuf` | Service metadata, name mapping, metadata centers |
| **Config center** | `dubbo-configcenter-zookeeper`, `-nacos`, `-apollo`, `-file` | External dynamic configuration sources |
| **Metrics / tracing** | `dubbo-metrics-api`, `-default`, `-prometheus`, `-otlp`, `dubbo-tracing` | Observability collection and export |
| **Spring Boot** | `dubbo-spring-boot-autoconfigure`, `-3-autoconfigure`, `-actuator`, `-starters/*` | Auto-config and the user-facing starters |
| **Plugins** | `dubbo-qos`, `dubbo-auth`, `dubbo-security`, `dubbo-spring-security`, `dubbo-filter-*`, `dubbo-rest-*`, `dubbo-mcp`, `dubbo-native`, `dubbo-compiler`, `dubbo-maven-plugin` | Operations port, auth, REST/OpenAPI, MCP, native image, codegen |
| **Compatibility** | `dubbo-compatible` | `com.alibaba.dubbo.*` shims for migrating from Dubbo 2.x |
| **Distribution** | `dubbo-distribution/*`, `dubbo-bom`, `dubbo-dependencies-bom` | Shaded "all" jar, BOMs |
| **Demo / test** | `dubbo-demo/*`, `dubbo-test/*` | Runnable examples and the test harness |

> [!TIP]
> When wiring a production app, depend on the **leaf** modules you need (one protocol, one registry, one serializer, the starters) rather than the umbrella `dubbo` artifact, to keep the classpath minimal and the attack surface small.

### 5.1 The `com.alibaba.dubbo.*` compatibility layer

`dubbo-compatible` ships deprecated shims under the old `com.alibaba.dubbo` package (e.g. `com.alibaba.dubbo.config.annotation.Service`, `com.alibaba.dubbo.remoting.Server`). They exist only to ease migration from Dubbo 2.x. **Do not use them in new code** — they are marked `@Deprecated` and delegate to the `org.apache.dubbo.*` equivalents.

---

## 6. The Invocation Pipeline

A consumer call traverses this sequence; understanding it is essential for debugging latency, retries, and routing.

1. **Proxy** — The `javassist` `ProxyFactory` generated a dynamic proxy for your interface. Calling a method produces an `RpcInvocation`.
2. **Cluster invoker** — The proxy's `Invoker` is a cluster invoker (default `FailoverClusterInvoker`). It owns fault tolerance.
3. **Directory** — Supplies the current candidate `Invoker` list (a `RegistryDirectory` for registry-backed references, kept live by registry pushes; a `StaticDirectory` for direct URLs).
4. **Router chain** — Filters the candidate list by condition/tag/script/affinity/mesh rules. Routers are dynamic — rules push from the config center.
5. **LoadBalance** — Picks one invoker from the survivors (default `RandomLoadBalance`).
6. **Filter chain** — `@Activate` filters run in order (validation, auth, access log, metrics, tracing, generic, …). This is where cross-cutting concerns live.
7. **Protocol invoker** — The protocol-specific invoker (`DubboInvoker`, Triple invoker) serializes and sends over the transport.
8. **Exchange + transport** — `HeaderExchanger` adds request/response correlation; Netty4 carries the bytes.

On the **provider** side the mirror image runs: transport → codec/serialization → exchange → protocol `Exporter` → provider-side filter chain → your implementation, dispatched on a configurable thread pool.

> [!NOTE]
> The `ZoneAwareClusterInvoker → FailoverClusterInvoker(RegistryDirectory)` nesting you see in logs is the multi-registry case: the zone-aware layer picks a registry, the failover layer picks a provider within it. Single-registry references collapse to one cluster layer.

### 6.1 Where retries happen — and the danger

`failover` retries at step 2 (cluster), re-running steps 3–8 on a different invoker. Because retries re-enter the full pipeline, a non-idempotent operation can execute more than once. The framework cannot know which methods are idempotent — **operators and developers must set `cluster=failfast` or `retries=0` for writes.** This is the most common production correctness pitfall.

---

## 7. Service Discovery Internals

### 7.1 Application-level discovery (the Dubbo 3 model)

In Dubbo 2, every interface registered one node per instance — a 100-instance app exposing 50 interfaces created 5,000 registry entries. Dubbo 3 registers **one node per application instance** and stores the **interface → application** mapping separately as metadata. The pieces:

- **`ServiceDiscovery`** — registers/queries application instances (`ServiceDiscoveryRegistryFactory`, registry name `service-discovery-registry`).
- **`ServiceNameMapping`** — resolves which application(s) provide a given interface, published at export time.
- **Metadata report** (`dubbo-metadata-report-*`) — stores per-revision service metadata (methods, parameters, config) so consumers can reconstruct the full provider URL from a lightweight instance registration.
- **Migration** (`dubbo-registry-api .../client/migration`) — lets a cluster run interface-level and application-level discovery simultaneously and shift between them without downtime, controlled by a migration rule.

### 7.2 Empty-protection

`enable-empty-protection` (on application and registry config) guards against a registry pushing an empty provider list (e.g. during a registry hiccup) and wiping a consumer's working address list. When enabled, an empty push is ignored and the last-known-good list is retained. **Keep this on in production.**

---

## 8. Requirements & Dependencies

### 8.1 Runtime requirements

| Requirement | Value |
|---|---|
| **JDK** | 8 minimum (source level 1.8); supported through 25 on the `3.3.x` line |
| **Build** | Maven 3.x; `dubbo-maven-plugin` for Protobuf/Triple stub generation |
| **Registry** | One of Zookeeper, Nacos (production); Multicast for dev only |
| **Optional infra** | Config center, metadata center (often the same Zookeeper/Nacos), Prometheus, Zipkin/OTLP collector |

### 8.2 Key third-party dependencies (from `dubbo-dependencies-bom`)

| Dependency | Version | Used by |
|---|---|---|
| Netty 4 | 4.x | Primary transport (`dubbo-remoting-netty4`) |
| Spring Framework | 5.3.39 (and legacy 4.3.30) | `dubbo-config-spring` |
| Spring Boot | 2.7.18 (3.x via `-3-autoconfigure`) | Starters |
| Curator | 5.9.0 | Zookeeper client |
| Zookeeper | 3.7.2 | Zookeeper registry |
| Nacos client | 2.5.2 | Nacos registry |
| Hessian | 4.0.66 | Default serialization |
| Fastjson2 | 2.0.62 | Alternative serialization |
| gRPC | 1.78.0 | Triple/gRPC interop |
| Jackson | 2.21.3 | JSON/REST |
| Micrometer / OpenTelemetry / Zipkin | (BOM-managed) | Metrics & tracing |

> [!CAUTION]
> Dubbo deliberately gives consumers control over transitive versions: the starters use `<exclusions>` heavily (e.g. excluding `spring-boot-starter-logging` in favor of log4j2, and Netty/logback from the Zookeeper client). When you upgrade Dubbo, re-verify your effective dependency tree (`mvn dependency:tree`) — version skew between Dubbo's expected client libraries (Curator, Nacos client, Netty) and yours is a frequent source of `NoSuchMethodError` at runtime.

---

## 9. Deployment

Dubbo deploys as part of your application — there is no Dubbo server to run. The deployment concerns are about **topology**, **process model**, and the **supporting infrastructure**.

### 9.1 Topology

```
        ┌─────────────┐         register/heartbeat        ┌──────────────┐
        │  Provider 1 │ ───────────────────────────────▶ │              │
        │  :20880(tcp)│                                    │   Registry   │
        │  :50051(tri)│ ◀── subscribe (provider list) ──── │  (ZK/Nacos)  │
        └─────────────┘                                    │              │
              ▲  ▲                                          │  + Config    │
   RPC call   │  │ heartbeat                                │  + Metadata  │
              │  └──────────────┐                           └──────────────┘
        ┌─────────────┐         │                                   ▲
        │  Consumer   │ ────────┘ discover + invoke                 │
        │             │ ──────────────────────────────────────────►│ publish metadata
        └─────────────┘
```

- **Providers** open one port per protocol (e.g. `20880` for `dubbo`, `50051` for `tri`), register in the registry, and publish metadata.
- **Consumers** subscribe, cache the address list locally (`dubbo.registry.file`), and call providers directly — the registry is **not** in the data path.
- The **registry/config/metadata** roles are often the same Zookeeper or Nacos cluster (`use-as-config-center`, `use-as-metadata-center`).

### 9.2 Containers & Kubernetes

- Expose each protocol port and the QOS port (`22222`).
- Wire the QOS HTTP probes (`/live`, `/ready`, `/startup`) into liveness/readiness/startup probes.
- For service mesh, set `mesh-enable: true` and use Triple; Dubbo resolves providers via Kubernetes DNS (`<svc>.<ns>.svc.cluster.local`) and lets Envoy/Istio route. The default cluster domain is `cluster.local` and mesh port `80`.
- The `dubbo-native` module provides GraalVM native-image hints and a sample `Dockerfile` (GraalVM CE) for AOT-compiled, fast-start, low-footprint deployments.

### 9.3 Graceful rollout

1. `offline` (or `offlineApp`) via QOS to deregister the instance — consumers stop routing to it within a refresh cycle.
2. Wait for in-flight calls to drain (`shutwait`).
3. Deploy the new version.
4. `online` to re-register.

> [!TIP]
> Combine the QOS `offline`/`online` commands with your orchestrator's `preStop` hook so rolling deploys drain cleanly without dropped requests.

---

## 10. Security

Dubbo's security surface spans the wire, the serializer, authentication, and the operations port. Harden all four.

### 10.1 The QOS operations port — the highest-risk surface

The QOS port (`22222`) can **`offline`, `shutdown`, `switchLogLevel`, and `invoke` arbitrary methods**. Treat it like a management interface:

- It binds **localhost-only** by default (`qos-accept-foreign-ip: false`). **Keep it that way.**
- If remote access is unavoidable, set `qos-accept-foreign-ip-whitelist` to explicit IPs and lower `qos-anonymous-access-permission-level` / restrict `qos-anonymous-allow-commands`.
- Never route the QOS port through an ingress or expose it publicly.

> [!CAUTION]
> Exposing the QOS port to an untrusted network is equivalent to handing out a remote shutdown and arbitrary-invoke button for your service. This is the #1 Dubbo misconfiguration to audit.

### 10.2 Serialization allow-listing

Java deserialization is a classic RCE vector. Dubbo defends with a **serialization security wrapper** and a class allow-list:

- `serialize-check-status` (`STRICT` / `WARN` / `DISABLE`) controls enforcement.
- `auto-trust-serialize-class` + `trust-serialize-class-level` govern which classes may be deserialized.
- `serializeCheckStatus` and `serializeWarnedClasses` QOS commands report what's being blocked or warned.

> [!WARNING]
> Two real serialization hazards from the issue tracker: (1) classes with a `writeReplace()` method can bypass the Hessian2 `Serializable` security check (#16287) — audit such classes; (2) Hessian2 may skip field-initialization on deserialization, yielding nulls/defaults (#15805). Prefer `fastjson2` for affected payloads and keep `serialize-check-status` at `STRICT` in production. Run with the allow-list enabled; do **not** set it to `DISABLE` to "fix" a deserialization error — fix the allow-list entry instead.

### 10.3 Authentication & transport security

- **Token auth** (`token = "true"` on a service) requires consumers to present a registry-distributed token; blocks unauthorized direct callers.
- **`dubbo-auth`** provides request signing/authentication filters; **`dubbo-spring-security`** / **`dubbo-spring6-security`** integrate Spring Security context propagation across RPC calls.
- **TLS** via `dubbo.ssl.*` (`SslConfig`) and `ssl-enabled` per protocol encrypts the wire; `dubbo.registry.secure` secures the registry connection.

### 10.4 Disclosure process

Security issues are reported through the Apache process documented in the repository's `SECURITY.md`. Do not file security bugs as public GitHub issues.

---

## 11. Operations

### 11.1 The QOS command catalog

Connect via `telnet localhost 22222` or HTTP (`curl http://localhost:22222/<command>`). Full command set (read directly from the `@Cmd` declarations):

| Category | Commands |
|---|---|
| **Inspection** | `ls`, `ps`, `pwd`, `cd`, `getConfig`, `getAddress`, `count`, `version`, `help`, `greeting` |
| **Service rotation** | `online`, `offline`, `onlineApp`, `offlineApp`, `onlineInterface`, `offlineInterface` |
| **Lifecycle** | `live`, `ready`, `startup`, `shutdown`, `gracefulShutdown`, `quit` |
| **Invocation** | `invoke`, `select` |
| **Metadata** | `publishMetadata`, `getOpenAPI` |
| **Logging** | `loggerInfo`, `switchLogger`, `switchLogLevel` |
| **Profiler** | `enableSimpleProfiler`, `enableDetailProfiler`, `disableSimpleProfiler`, `disableDetailProfiler`, `setProfilerWarnPercent` |
| **Router debug** | `enableRouterSnapshot`, `disableRouterSnapshot`, `getRouterSnapshot`, `getEnabledRouterSnapshot`, `getRecentRouterSnapshot` |
| **Serialization** | `serializeCheckStatus`, `serializeWarnedClasses` |
| **Metrics** | `metrics_default` |

### 11.2 Runtime diagnosis playbook

| Symptom | QOS commands to run |
|---|---|
| "Consumer can't find provider" | `ls` (both sides), `getAddress <service>` |
| "Calls routed to the wrong instance" | `enableRouterSnapshot` then `getRecentRouterSnapshot` |
| "Some calls are slow" | `enableDetailProfiler`, `setProfilerWarnPercent`, then inspect logs |
| "Need more log detail without restart" | `switchLogLevel DEBUG`, revert with `switchLogLevel INFO` |
| "Deserialization blocked" | `serializeCheckStatus`, `serializeWarnedClasses` |
| "Draining for deploy" | `offline`, wait, deploy, `online` |

### 11.3 Logging

Dubbo abstracts logging via `dubbo.application.logger` (`slf4j`, `log4j2`, `jcl`, …). The demos use log4j2 and exclude `spring-boot-starter-logging`. Error logs carry **stable error codes** (e.g. `CLUSTER_NO_VALID_PROVIDER`, `CONFIG_FAILED_SHUTDOWN_HOOK`) defined in `LoggerCodeConstants` — grep logs by code for precise diagnosis.

---

## 12. Observability

Dubbo ships first-class metrics and tracing, enabled through starters and `dubbo.metrics.*` / `dubbo.tracing.*`.

- **Metrics** (`dubbo-metrics-*`): Micrometer-based, exportable to Prometheus (`dubbo-metrics-prometheus`) or OTLP (`dubbo-metrics-otlp`). Covers RPC QPS/RT/error rates, thread pools, registry events, and JVM metrics. Add `dubbo-observability-spring-boot-starter`.
- **Tracing** (`dubbo-tracing`): Micrometer Tracing bridges to **Brave/Zipkin** or **OpenTelemetry** (OTLP or Zipkin export). Starters: `dubbo-tracing-brave-zipkin-spring-boot-starter`, `dubbo-tracing-otel-zipkin-spring-boot-starter`, `dubbo-tracing-otel-otlp-spring-boot-starter`. Configure sampling with `dubbo.tracing.sampling.probability`.
- **Actuator**: `dubbo-spring-boot-actuator` exposes Dubbo health/info endpoints.

```yaml
dubbo:
  metrics:
    protocol: prometheus
    enable-jvm-metrics: true
  tracing:
    enabled: true
    sampling: { probability: 0.1 }
    tracing-exporter:
      otlp-config: { endpoint: http://otel-collector:4318/v1/traces }
```

---

## 13. Performance Tuning

| Lever | Where | Guidance |
|---|---|---|
| **Protocol** | `dubbo.protocol.name` | `dubbo` (TCP) for max throughput between Java services; `tri` for HTTP/2/mesh |
| **Thread pool** | `dubbo.protocol.threadpool` (`fixed`/`cached`/`limited`/`eager`) + `threads`, `corethreads`, `queues` | Size to provider workload; `eager` grows before queuing |
| **Thread isolation** | `dubbo.application.executor-management-mode: isolation` | Give heavy services their own pool so one can't starve others |
| **IO threads** | `dubbo.protocol.iothreads` | Netty event-loop sizing (default = cores) |
| **Connections** | `connections` / `dubbo.consumer.shareconnections` | Share or dedicate TCP connections per reference |
| **Payload cap** | `dubbo.protocol.payload` | Max message size; raise carefully (memory) |
| **Serialization** | `dubbo.protocol.serialization` | `hessian2` default; `fastjson2` can be faster for some shapes |
| **Load balance** | `loadbalance` | `leastactive`/`shortestresponse` smooth out slow nodes; `adaptive` is load-aware |
| **Concurrency limits** | `actives` (consumer), `executes` (provider) | Bound in-flight calls to prevent overload/OOM (see #16294) |

> [!TIP]
> Use the QOS profiler (`enableDetailProfiler` + `setProfilerWarnPercent`) in a canary to find slow methods before tuning blindly. Pair `leastactive` load balancing with `actives` limits so a degraded provider sheds load instead of accumulating it.

---

## 14. Known Issues & Tech Debt

Drawn from the project's own issue tracker at this commit — useful context for operators choosing versions.

| Issue | Area | Operator action |
|---|---|---|
| #16295 / #16296 | JDK 25 + ServiceLoader/Gson `NoClassDefFoundError` | Use a `3.3.x` build with JDK 25 support, or pin JDK 21; ensure Gson present if needed |
| #16294 | Large `consumer.timeout` → OOM | Keep timeouts realistic; bound concurrency with `actives`/`executes` |
| #16178 | Provider start fails when ZK (weak dep) is down | `dubbo.registry.check: false` |
| #15805 | Hessian2 skips field init on deserialize | Set fields explicitly or use `fastjson2` |
| #16287 | `writeReplace()` bypasses Hessian2 `Serializable` security check | Audit such classes; keep allow-list `STRICT` |
| #16308 | `NetUtils.matchIpExpression` mis-parses IPv4+port | Separate host/port in config; upgrade |
| #16117 | `CompositeInputStream.close()` may leak streams | Tracking; upgrade when fixed |
| #16214 | Spring Boot 4 compatibility not yet available | Stay on supported Spring Boot 2.7/3.x lines |

**Structural debt worth knowing:** the `com.alibaba.dubbo.*` compatibility layer is dead weight for new projects (don't depend on it); the source level is still 1.8, so the codebase cannot use modern Java language features internally even while supporting JDK 25 at runtime; and the SPI metadata files (`META-INF/dubbo/internal/*`) are the source of truth for extension wiring but are easy to miss when reviewing — a missing line silently disables an extension.

---

## 15. Contributing

- **Build:** `mvn clean install` (Maven multi-module). The `${revision}` property sets the version across all modules — do not hard-code versions in child POMs.
- **Code style:** enforced via the `codestyle` module and the Maven compiler config; run the formatter before submitting.
- **Adding an extension:** implement the `@SPI` interface, add a `META-INF/dubbo/<InterfaceName>` mapping line, and a test. Do not add conditionals to core — the SPI is the extension seam.
- **Spring config metadata:** config fields live in `dubbo-common`'s `org.apache.dubbo.config.*` classes; their **Javadoc** is harvested to generate `spring-configuration-metadata.json` for IDE auto-completion (a dedicated compatible module copies the classes and substitutes `@NestedConfigurationProperty`). Follow the Javadoc conventions ("The xxx" for nouns, "Whether to xxx, default value is `true`" for booleans) so the generated metadata is correct.
- **Conduct & security:** see `CODE_OF_CONDUCT.md`, `CONTRIBUTING.md`, and `SECURITY.md` at the repo root.

> [!NOTE]
> Because Java AST tooling does not parse this codebase for extraction, the authoritative reference for any contributor question is the source itself — start from the `@SPI` interface for the layer you're changing and the `META-INF/dubbo/internal/<interface>` file that wires its implementations.

---

## 16. Quick Reference

### Default extensions

| Layer | Default | Alternatives |
|---|---|---|
| Protocol | `dubbo` | `tri`, `grpc`, `rest2`, `injvm`, `mock` |
| Cluster | `failover` | `failfast`, `failsafe`, `failback`, `forking`, `broadcast`, `available`, `mergeable`, `zone-aware` |
| LoadBalance | `random` | `roundrobin`, `leastactive`, `shortestresponse`, `consistenthash`, `adaptive` |
| Serialization | `hessian2` | `fastjson2` |
| Proxy factory | `javassist` | `jdk` |
| Registry | (none) | `zookeeper`, `nacos`, `multicast`, `multiple`, `service-discovery-registry` |

### Default ports

| Port | Purpose |
|---|---|
| `20880` | `dubbo` (TCP) protocol — conventional default |
| `50051` | Triple (`tri`) — conventional example port |
| `22222` | QOS operations (telnet + HTTP) |

### Scope-model hierarchy

```
FrameworkModel  →  ApplicationModel  →  ModuleModel
(JVM/global)       (per application)     (per module's services/refs)
```

### Configuration precedence (highest first)

`-D` system properties → external config center → environment variables → `application.yml`/`dubbo.properties` → annotation/XML attributes → built-in defaults.

### Essential QOS commands

`ls` · `getAddress` · `online` / `offline` · `gracefulShutdown` · `enableDetailProfiler` · `getRecentRouterSnapshot` · `switchLogLevel` · `serializeCheckStatus`

---

_Generated June 2026 from apache/dubbo at ee9c190. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._