Milvus Admin Guide

This guide is the operational and architectural reference for Milvus, the open-source distributed vector database. It targets administrators, SREs, platform engineers, and architects who deploy, configure, secure, monitor, and extend Milvus in production. It assumes familiarity with the user-facing concepts (collections, vectors, indexes, search) covered in the companion User Guide.

Milvus is written in Go (the distributed control and data planes) and C++ (the internal/core vector search and storage engine, "Segcore" and the Knowhere index library). It is designed to be cloud-native and K8s-native, separating compute from storage so that read and write capacity scale independently. Source references in this guide point at the Milvus codebase at commit d9bbbc1.

1. Executive Summary

The defining design choice is storage/compute disaggregation built on a shared log. All durable state lives in external systems (etcd for metadata, object storage for data, a WAL for the recent mutation stream). Milvus's own processes are stateless and restartable, which is what enables horizontal scaling and fast failure recovery.

Important

As of the 2.6 line (this codebase, image milvusdb/milvus:v3.0-beta), the streaming service is enabled by default and the three legacy coordinators (root/data/query) are unified into a single MixCoord process. Plan deployments around MixCoord + workers + StreamingNode, not the older per-coordinator model.

2. Architecture

2.1 Component model

A Milvus deployment is composed of these roles, each started by the milvus binary via milvus run <role> (cmd/roles/roles.go, cmd/components/*):

                         ┌──────────────────────────────┐
   SDK / REST clients ──▶│            Proxy             │  stateless gateway (gRPC+REST :19530)
                         └──────────────┬───────────────┘
                                        │
              ┌─────────────────────────┼─────────────────────────┐
              ▼                         ▼                         ▼
      ┌───────────────┐        ┌────────────────┐        ┌────────────────┐
      │   MixCoord    │        │  QueryNode(s)  │        │  DataNode(s)   │
      │ root+data+    │        │  load segments │        │  consume WAL,  │
      │ query coord   │        │  run searches  │        │  flush, compact│
      └───────┬───────┘        └───────┬────────┘        └───────┬────────┘
              │                        │                         │
              │                ┌───────┴─────────┐               │
              │                │  StreamingNode  │  WAL append/scan, time-tick
              │                └───────┬─────────┘               │
   ┌──────────┴───────────┬───────────┴─────────────┬───────────┴────────┐
   ▼                      ▼                          ▼                    ▼
 etcd                 Message queue / WAL        Object storage      (optional GPU)
 metadata +           Woodpecker | Pulsar |      MinIO | S3 | GCS |
 service discovery    Kafka | RocksMQ            Azure Blob

Note

MixCoord internally still contains the RootCoord (DDL/DCL), DataCoord (data/segment management), and QueryCoord (query topology) subsystems — their configuration sections (rootCoord.*, dataCoord.*, queryCoord.*) remain separate even though they run in one process. In a fully disaggregated cluster you may still run them as separate processes via the mixture server type with role flags.

2.2 The milvus run role matrix

cmd/milvus/util.go:GetMilvusRoles maps a server type to the set of enabled components:

Roles are also selectable by environment variable (ENABLE_PROXY, ENABLE_QUERY_NODE, ENABLE_DATA_NODE, ENABLE_STREAMING_NODE, ENABLE_ROOT_COORD, ENABLE_QUERY_COORD, ENABLE_DATA_COORD, ENABLE_CDC).

2.3 The storage layers

Milvus separates three kinds of state into three external systems:

1. Metadata & coordination → etcd. Collection schemas, segment metadata, index metadata, node sessions, and service discovery live in etcd under etcd.rootPath (default by-dev). TiKV is an alternative metadata store for very large metadata (metastore.type: tikv), but etcd is still required for service discovery.
2. Durable data → object storage. Sealed segment binlogs, delta logs, statistics, and index files are written to MinIO/S3/GCS/Azure under minio.*. This is the system of record for data.
3. Recent mutations → WAL / message queue. New inserts and deletes flow through a write-ahead log before they are flushed to object storage. The WAL implementation is pluggable: Woodpecker (native, recommended for new installs), Pulsar, Kafka, or RocksMQ (standalone only).

Tip

For new deployments the project recommends Woodpecker as the WAL/MQ for better performance, simpler operations, and lower cost. RocksMQ is only valid in standalone mode; cluster mode defaults to Pulsar when mq.type is left at default.

2.4 Data lifecycle and segments

Data moves through growing → sealed → indexed segments:

• Writes land in growing segments (in memory, also mmappable) via the WAL.
• DataCoord seals a segment when it reaches size/age thresholds (dataCoord.segment.maxSize, sealProportion, maxLife, maxIdleTime, or the sealPolicy.channel.* watermarks).
• Sealed segments are flushed to object storage and indexed (vector and scalar indexes are built by DataNode/index tasks).
• Compaction merges small segments, applies deletes, and runs clustering compaction (dataCoord.compaction.*). Garbage collection removes orphaned files (dataCoord.gc.*).
• QueryNodes load sealed+indexed segments to serve search; a delegator forwards streaming deletes and growing data.

2.5 The C++ core (Segcore + Knowhere)

The internal/core/src tree is the C++ engine, reached from Go via cgo wrappers (internal/util/segcore, indexcgowrapper, analyzecgowrapper):

• Segcore — in-segment storage, chunked columnar layout, expression evaluation, and the search/query operators.
• Knowhere — the vector index library (knowhere.* config) implementing FLAT, IVF*, HNSW, SCANN, DISKANN, and GPU (CAGRA) indexes with CPU SIMD acceleration (common.simdType: auto/avx512/avx2/avx/sse4_2).
• Tiered storage — queryNode.segcore.tieredStorage.* controls memory/disk cache warmup and eviction watermarks for hot/cold data.

3. Requirements

3.1 Build requirements

From README.md / DEVELOPMENT.md, building Milvus from source needs (Linux/Ubuntu 20.04+):

Go:     >= 1.21   (go.mod pins go 1.25.x)
CMake:  >= 3.26.4 && < 4
GCC:    >= 11     (Linux)  /  llvm >= 15 (macOS)
Python: > 3.8 and <= 3.11

git clone https://github.com/milvus-io/milvus.git
cd milvus
./scripts/install_deps.sh
make

3.2 Runtime dependencies

Object storage may instead be AWS S3, Google Cloud Storage (gcpnative), Azure Blob, Aliyun OSS, Tencent COS, or Huawei OBS (see minio.cloudProvider and pkg/objectstorage/*).

3.3 Sizing guidance

• Memory is the primary constraint for QueryNodes — loaded segments and indexes are held in RAM (or mmap/tiered cache). Size query nodes to your loaded dataset plus index overhead.
• DataNodes are CPU/memory bound during flush, compaction, and index building.
• Object storage must hold all segment, index, delta, and stats files; plan for several times the raw vector size after indexing and replication.

4. Configuration

Milvus is configured by configs/milvus.yaml (1,500+ keys). Configuration is layered: YAML file → environment variables (a subset, e.g. ETCD_ENDPOINTS, MINIO_ADDRESS, MINIO_ACCESS_KEY_ID) → runtime overrides via etcd (some keys are hot-reloadable; the paramtable watches etcd and applies changes live). Some keys are immutable and persisted at first startup (e.g. the selected WAL name).

Note

Many keys preferentially read an environment variable when present — etcd.endpoints reads ETCD_ENDPOINTS, minio.address reads MINIO_ADDRESS, and the MinIO keys read MINIO_ACCESS_KEY_ID / MINIO_SECRET_ACCESS_KEY. This is how the Docker Compose files inject infrastructure addresses.

4.1 Metadata store — etcd / metastore / tikv

Caution

Changing etcd.rootPath, metaSubPath, kvSubPath, or the msgChannel prefixes after a cluster has data will make existing data unreadable. Set them once, before first start.

4.2 Object storage — minio / localStorage

Tip

In production prefer useIAM: true with instance/workload identity over static accessKeyID/secretAccessKey. Static keys in milvus.yaml are a credential-leak risk.

4.3 Message queue / WAL — mq / woodpecker / pulsar / kafka / rocksmq

Priority when mq.type: default: standalone → rocksmq > Pulsar > Kafka > Woodpecker; cluster → Pulsar > Kafka > Woodpecker (RocksMQ unsupported in cluster).

4.4 Coordinator and node configuration

Each role has its own section. Key knobs (selected — see configs/milvus.yaml for the full set):

rootCoord (DDL/DCL): maxPartitionNum (1024), maxDatabaseNum (64), dmlChannelNum (16), minSegmentSizeToEnableIndex (1024), enableActiveStandby (false), port 22125.

proxy (gateway): maxNameLength (255), maxFieldNum (64), maxVectorFieldNum (10), maxShardNum (16), maxDimension (32768), maxTaskNum (1024), maxConnectionNum (10000), http.enabled (true), http.enableWebUI (true), accessLog.*, port 19530, internalPort 19529, grpc.serverMaxRecvSize (64 MB), grpc.serverMaxSendSize (256 MB).

queryCoord (topology/balance): autoBalance (true), balancer (ScoreBasedBalancer), overloadedMemoryThresholdPercentage (90), balanceIntervalSeconds (60), loadTimeoutSeconds (600), enableStoppingBalance (true), port 19531.

queryNode (search execution):

• Segcore: segcore.chunkRows (128), knowhereThreadPoolNumRatio (4), interimIndex.enableIndex (true), interimIndex.nlist/nprobe.
• Memory/mmap: mmap.vectorField (true), mmap.vectorIndex (false), cache.readAheadPolicy (willneed), loadMemoryUsageFactor (1), enableDisk (false), maxDiskUsagePercentage (95).
• Tiered storage: segcore.tieredStorage.warmup.* (sync/async/disable), memoryLowWatermarkRatio (0.75) / memoryHighWatermarkRatio (0.8), evictionEnabled (false).
• Scheduler: scheduler.maxReadConcurrentRatio, scheduler.scheduleReadPolicy.name (fifo or user-task-polling). port 21123.

dataCoord (data lifecycle):

• Segments: segment.maxSize (1024 MB), diskSegmentMaxSize (2048 MB), sealProportion (0.12), maxLife (86400 s), maxIdleTime (600 s).
• Compaction: enableCompaction (true), compaction.enableAutoCompaction (true), compaction.clustering.enable (true), compaction.taskPrioritizer (level), L0/mix/single/clustering trigger intervals.
• GC: enableGarbageCollection (true), gc.interval (3600 s), gc.missingTolerance (86400 s), gc.dropTolerance (10800 s). port 13333.

dataNode (flush/compaction):

• Sync: dataSync.flowGraph.maxQueueLength (16), memory.forceSyncEnable (true), memory.forceSyncWatermark (0.5).
• Segment buffer: segment.insertBufSize (16 MB), segment.deleteBufBytes (16 MB), segment.syncPeriod (600 s).
• Slots: slot.slotCap (16). Storage format: storage.format (parquet), storage.deltalog (json). gracefulStopTimeout (1800 s). port 21124.

streamingNode / streaming (WAL service): streamingNode.port 22222; streaming.walBalancer.* (balance policy vchannelFair), streaming.flush.* (memory/growing-segment watermarks), streaming.walRecovery.*, streaming.txn.*, streaming.replication.*.

4.5 Common, GPU, Knowhere

common holds cross-cutting settings: common.security.* (see Security), common.gracefulStopTimeout (1800 s), common.simdType (auto), common.storageType (remote), thread-pool coefficients (threadCoreCoefficient.*), DiskIndex tuning (common.DiskIndex.*), bloom-filter settings, and JSON-shredding toggles.

gpu (GPU builds only): gpu.initMemSize (2048 MB), gpu.maxMemSize (4096 MB), gpu.overloadedMemoryThresholdPercentage (95).

knowhere: per-index-type default parameters (AISAQ, DISKANN build/search params) auto-populated when knowhere.enable is true.

4.6 Logging, gRPC, tracing

Log level is hot-reloadable — the server watches log.level in etcd and applies changes without restart (cmd/roles/roles.go:setupLogger). Tracer config is likewise re-applied on change.

5. Security

Caution

A default Milvus install is insecure by design for ease of first use: authorization is off (common.security.authorizationEnabled: false) and the root user has the well-known password Milvus. Before exposing Milvus to any shared or untrusted network, enable auth, change the root password, and enable TLS.

5.1 Authentication

• common.security.authorizationEnabled — set true to require credentials on every request.
• common.security.defaultRootPassword (Milvus) — change this; the root account is the bootstrap admin. (Max length 72 chars.)
• common.security.superUsers — users that bypass some checks (e.g. the old-password check on credential update).
• common.security.rootShouldBindRole — require root to bind a role when auth is on.
• common.security.enablePublicPrivilege (true) — whether a default public privilege exists.

Clients authenticate with username/password (sent base64-encoded in the authorization metadata header) or an API key (client/milvusclient/client.go:parseAuthentication).

5.2 RBAC

Milvus implements role-based access control backed by Casbin (github.com/casbin/casbin). Administrators manage users, roles, privileges, and privilege groups (SDK methods in client/milvusclient/rbac.go and rbac_v2.go):

• Users — CreateUser, UpdatePassword, DropUser, ListUsers, DescribeUser.
• Roles — CreateRole, DropRole, GrantRole, RevokeRole, DescribeRole, ListRoles.
• Privileges (v2) — GrantPrivilegeV2, RevokePrivilegeV2 grant a privilege on an object (collection/database/etc.) to a role.
• Privilege groups — CreatePrivilegeGroup, AddPrivilegesToGroup, RemovePrivilegesFromGroup, ListPrivilegeGroups bundle privileges for reuse.
• Backup/restore — BackupRBAC / RestoreRBAC export and import the entire RBAC model (users, roles, grants, privilege groups) — use these before upgrades or for DR.

proxy.resolveAliasForPrivilege (true) controls whether aliases resolve to real collection names during privilege checks.

5.3 TLS

Two independent TLS scopes:

• External TLS (tls.*) secures client↔Proxy traffic: serverPemPath, serverKeyPath, caPemPath. Enable on the client with EnableTLSAuth.
• Internal TLS (internaltls.* + common.security.internaltlsEnabled) secures intra-cluster gRPC between components: serverPemPath, serverKeyPath, caPemPath, and an sni matching the certificate. common.security.tlsMode selects the TLS mode (0/1/2 for off/one-way/two-way).
• etcd, MinIO, and Kafka each have their own TLS blocks (etcd.ssl, minio.ssl, kafka.ssl).

Note

A FIPS-validated build is available: the binary can enable OpenSSL FIPS / Go BoringCrypto (cmd/milvus/boring_*.go, pkg/util/fips). The startup banner reports Milvus FIPS in Go: BoringCrypto <bool>.

5.4 The /expr debug endpoint

common.security.exprEnabled (default false) gates the /expr HTTP endpoint used for live expression debugging. When enabled it is restricted to the root user via HTTP Basic Auth on Proxy nodes. Leave it disabled in production.

5.5 Quotas and rate limits

quotaAndLimits.* is both a stability mechanism and a multi-tenant fairness control. By default Milvus enables time-tick protection, memory protection, and disk-quota protection; you can additionally enable DML/DDL/DQL rate limits.

Warning

When memory or disk water marks are crossed, Milvus throttles or rejects DML to protect the cluster. "Quota exceeded" / write-denied errors usually mean a node is near its memory high-water mark or the disk quota is hit — scale out or raise the quota rather than retrying blindly.

6. Deployment

6.1 Standalone (Docker Compose)

Use deployments/docker/standalone/docker-compose.yml (see the User Guide for the full file). It launches milvus-etcd, milvus-minio, and milvus-standalone (milvus run standalone), publishing 19530 (API) and 9091 (health/metrics/Web UI). A GPU variant lives at deployments/docker/gpu/standalone/.

sudo docker compose up -d
curl -f http://localhost:9091/healthz   # -> OK

6.2 Distributed cluster

For production scale, run roles as independent, separately scaled deployments on Kubernetes (the official Milvus Helm chart / Milvus Operator; this repository provides reference Ansible roles under deployments/docker/cluster-distributed-deployment/). Typical topology:

• Multiple Proxies behind a load balancer (stateless; scale for connection/throughput).
• MixCoord with enableActiveStandby for HA (one active, standbys ready).
• QueryNodes scaled for read/memory (each holds loaded segments).
• DataNodes scaled for write/compaction/index throughput.
• StreamingNodes for the WAL service.
• External etcd (3+ nodes), object storage, and Pulsar/Kafka/Woodpecker.

Tip

Scale QueryNodes for read-heavy or large-resident-data workloads and DataNodes for write-heavy/ingest workloads — this independent scaling is the main reason to choose the cluster topology over standalone.

6.3 High availability

• Coordinators support active-standby (*.enableActiveStandby): standbys take over via etcd session/lease on failure.
• Workers are stateless; losing one triggers QueryCoord auto-balance to redistribute segments.
• The WAL plus object storage make data durable independent of any Milvus process; recovery replays from the WAL and reloads from object storage.
• Replicas (multiple loaded copies of segments across query nodes) increase read throughput and fault tolerance — manage with resource groups and TransferReplica.

6.4 Resource groups

Resource groups partition QueryNodes into pools so you can isolate workloads/tenants (client/milvusclient/resource_group.go): CreateResourceGroup, UpdateResourceGroup, TransferReplica, DescribeReplica. quotaAndLimits.limits.maxResourceGroupNumOfQueryNode (1024) bounds the count.

7. Operations

7.1 Starting and stopping

milvus run standalone            # or: run mixcoord / proxy / querynode / datanode / streamingnode
milvus stop standalone           # graceful stop (sends SIGTERM via PID file)

Each role honors a gracefulStopTimeout; on timeout the process dumps goroutine/heap/block/ mutex pprof profiles to ProfileCfg.PprofPath and force-exits (cmd/components/util.go).

7.2 Health and readiness

• GET /healthz on 9091 — liveness.
• Component readiness is tracked internally; the Web UI and GetComponentStates expose per-role health (Healthy/Abnormal/StandBy).

7.3 Monitoring

• Prometheus metrics at :9091/metrics (Milvus registry) and /metrics_default (Go runtime). Build info, per-role metrics, meta/msgstream/storage metrics are registered at startup (cmd/roles/roles.go).
• Grafana dashboards are provided under deployments/monitor/.
• OpenTelemetry tracing via trace.exporter (jaeger/otlp); sampling via trace.sampleFraction.
• Access logs (proxy.accessLog.*) record every request with method, status, user, latency, and trace ID; optionally uploaded to object storage.
• Slow queries are flagged when they exceed proxy.slowQuerySpanInSeconds (1 s) and shown in the Web UI.

7.4 The Web UI

Enabled by proxy.http.enableWebUI (default true) on :9091/webui — inspect collections, segment distribution, query-node balance, slow queries, and component health.

7.5 Consistency check (mck)

milvus mck run scans etcd and object storage for orphaned task/segment references and reports invalid tasks, segments, and missing object-storage paths; milvus mck cleanTrash removes the orphaned entries (with backup to an mck-trash prefix). Flags: -etcdIp, -etcdRootPath, -minioAddress, -minioUsername, -minioPassword, -minioUseSSL, -minioBucketName.

Warning

mck mutates metadata. Run it against a quiesced cluster and keep the etcd backup it creates. It is a recovery tool, not routine maintenance.

7.6 Backup and restore

• RBAC: BackupRBAC / RestoreRBAC (SDK) export/import the full access-control model.
• Snapshots: CreateSnapshot / RestoreSnapshot capture and restore point-in-time collection data; dataCoord.snapshot.* governs retention and compaction protection.
• Data: because data lives in object storage and metadata in etcd, a consistent backup pairs an etcd snapshot with the object-storage bucket. The community Milvus Backup tool automates this.

7.7 Upgrades and migration

cmd/tools/migration performs metadata migrations between Milvus versions (deployments/migrate-meta/). DataCoord can rebuild segment indexes to a target index-engine version (dataCoord.autoUpgradeSegmentIndex, targetVecIndexVersion, forceRebuildSegmentIndex). Always back up etcd before a version upgrade and follow the official rolling-upgrade procedure for clusters.

7.8 Change data capture (CDC) and replication

The cdc role and streaming.replication.* settings support replicating data between clusters (internal/cdc/*). SDK surface: GetReplicateInfo, CreateReplicateStream, UpdateReplicateConfiguration. Use for cross-region DR or migration.

8. Internals (Developer Reference)

8.1 Repository layout

8.2 Process orchestration

cmd/roles/roles.go:Run starts enabled components concurrently via conc.Future, waits for all to be ready, registers them with the health subsystem and Prometheus, wires signal handling for graceful shutdown, and tears down in order (coordinators → nodes → proxy → shared etcd client). Config changes propagate through paramtable watchers on etcd.

8.3 API surface

The wire API is the gRPC milvuspb.MilvusService (defined in the external milvus-proto repository) plus the REST v2 layer (internal/distributed/proxy/httpserver, paths /v2/vectordb/<group>/<action>). The extraction TSVs only see test handlers; the real API is the proto service and the SDK methods enumerated in the User Guide.

Note

Because the protobuf service definitions live in the separate milvus-io/milvus-proto repository (vendored as github.com/milvus-io/milvus-proto/go-api/v3), the gRPC method list is not present in this repository's source. The authoritative client-visible surface is the Go SDK in client/ (and the equivalent pymilvus methods).

8.4 Key dependencies

From go.mod: go.etcd.io/etcd (metadata), minio-go + cloud SDKs (storage), apache/pulsar-client-go (MQ), apache/arrow/go (columnar), antlr4-go (search-expression parser), gin-gonic/gin (REST), casbin (RBAC), prometheus/client_golang (metrics), go.opentelemetry.io/otel (tracing), grpc, tikv/client-go, cockroachdb/errors.

8.5 Build and test

make builds the Go binaries and C++ core. make verifiers runs linters; make unittest runs unit tests. The dev environment is reproduced by the root docker-compose.yml (builder image plus etcd/pulsar/minio/azurite/jaeger). Client linting uses a strict golangci-lint config (client/.golangci.yml) that, for example, forbids the stdlib errors package in favor of cockroachdb/errors.

9. Code Quality and Conventions

• Error handling uses a centralized merr package (pkg/util/merr) producing a rich, catalogued error set (2,000+ messages) with codes — this is what surfaces to clients and feeds troubleshooting.
• Config access is centralized in paramtable; code reads typed accessors (.GetAsBool(), .GetAsInt(), .GetAsDuration()) rather than parsing YAML directly, enabling hot reload and type checking (common.enableConfigParamTypeCheck).
• Concurrency uses conc.Future/pools and structured goroutine management.
• cgo boundaries are isolated in *cgowrapper packages with explicit lifetime management for the C++ core.
• Linters are strict and enforced in CI (make verifiers, ruleguard custom rules).

10. Potential Improvements / Operational Watch-list

These are observations for operators and contributors, drawn from configuration and code:

• Deprecated knobs remain in milvus.yaml (e.g. queryNode.cache.memoryLimit, queryNode.segcore.multipleChunkedEnable, dataCoord.compaction.maxParallelTaskNum). Audit your config against the current defaults on upgrade rather than copying an old file forward.
• Storage format is evolving — dataNode.storage.format supports parquet (default) and vortex; deltalog supports json/parquet; Storage V2 (common.storage.stv2.*) and useLoonFFI are newer paths. Pin and test the format before large migrations.
• WAL choice matters — RocksMQ does not work in cluster mode; the project recommends Woodpecker for new installs. Migrating an existing cluster's WAL backend is non-trivial.
• Default credentials and disabled auth are the single biggest production risk; secure them before any network exposure.
• Memory-bound query nodes — without tiered storage/mmap tuning, large loaded collections can exhaust RAM and trigger write throttling cluster-wide. Tune queryNode.segcore.tieredStorage.* and queryNode.mmap.*, and use resource groups for isolation.

11. Quick Reference

11.1 Default ports

11.2 Essential commands

milvus run standalone                 # start all-in-one
milvus run mixcoord                    # start unified coordinator
milvus run proxy|querynode|datanode|streamingnode
milvus stop <server type>             # graceful stop
milvus mck run -etcdIp <ip> ...       # consistency check
milvus mck cleanTrash                  # clean orphaned data
curl -f http://localhost:9091/healthz  # health probe

11.3 Critical config keys (change before first start)

11.4 External dependency checklist

• [ ] etcd reachable at etcd.endpoints (3+ nodes for HA)
• [ ] Object storage reachable at minio.address, bucket creatable/writable
• [ ] WAL/MQ reachable (Pulsar/Kafka) or Woodpecker storage configured
• [ ] Auth enabled and root password changed
• [ ] TLS configured (external + internal)
• [ ] Prometheus scraping :9091/metrics; Grafana dashboards loaded
• [ ] Backups configured (etcd snapshot + object storage + RBAC export)