This guide is the operational and architectural reference for Kong Gateway 3.10.0 — the open-source, cloud-native API/LLM/MCP gateway built on OpenResty (NGINX + LuaJIT). It is written for administrators, SREs, platform engineers, and architects who deploy, secure, tune, operate, and extend Kong, and for contributors who need to understand the codebase.
For day-to-day usage (proxying APIs, configuring entities, using plugins), see the companion User Guide. This document is intentionally exhaustive: it covers the full configuration surface, every deployment topology, the internal architecture, the plugin development model, the database/migration system, and the security model.
Note
Everything here is derived directly from the Kong source at commit
6e6c302(version 3.10.0, NGINX dependency1.27.1.2, PostgreSQL minimum9.5). Where a capability is Enterprise-only or Konnect-only, it is labeled; the focus is the open-source distribution.
Table of Contents
- Executive Summary
- Architecture
- Deployment Topologies
- System Requirements
- Configuration Reference
- Operations
- Security
- Performance Tuning
- The Data Model
- Database & Migrations
- Repository Structure & Internals
- Plugin Development
- Observability
- Testing Strategy
- Potential Improvements & Tech Debt
- Contributing
- Quick Reference
1. Executive Summary
Kong Gateway is a reverse proxy and policy-enforcement layer that runs in front of your APIs and microservices. It is implemented in Lua on the OpenResty platform, which embeds LuaJIT inside NGINX. This gives Kong NGINX's proven networking core (event-driven, C-level performance) plus the flexibility of a scripting layer that can make routing and policy decisions per request.
What it provides:
- A declarative, data-driven policy model: you describe routing and behavior as entities (Services, Routes, Plugins, Consumers, Upstreams), and Kong enforces them on live traffic with no code changes to your backends.
- A runtime-extensible plugin system: 44 bundled plugins plus the ability to write custom plugins in Lua, Go, JavaScript, Python, or WebAssembly.
- Three deployment topologies: traditional (with PostgreSQL), DB-less (declarative, in-memory), and hybrid (control-plane/data-plane split over mTLS).
- An AI Gateway: a universal LLM API and prompt-governance plugins for AI/MCP traffic.
Key architectural facts an operator must internalize:
| Fact | Implication |
|---|---|
| Kong = OpenResty + Lua | Performance is NGINX-class; extensions run in LuaJIT per request. |
| Config lives as entities | Stored in PostgreSQL (traditional), in-memory LMDB (DB-less), or pushed CP→DP (hybrid). |
| Config is cached in shared memory | Changes propagate within db_update_frequency (default 5s) across nodes. |
Every setting has a KONG_-prefixed env var |
Twelve-factor friendly; env vars override the config file. |
| The Admin API is fully privileged | It must never be exposed publicly; securing it is the #1 hardening task. |
The router defaults to traditional_compatible |
Fast expression engine with the classic config interface; supports incremental rebuilds. |
2. Architecture
2.1 High-level component view
┌──────────────────────────────────────────────────────┐
│ Kong Gateway node │
│ (OpenResty: NGINX + LuaJIT) │
clients ──proxy───────►│ proxy_listen :8000/:8443 ┌───────────────────────┐ │──► upstreams
│ │ Runloop (handler.lua)│ │ (your services)
admins ──Admin API────►│ admin_listen :8001/:8444 │ • router │ │
│ │ • plugins iterator │ │
monitors ─Status API──►│ status_listen :8007 │ • balancer │ │
│ │ • DNS client │ │
data planes ─cluster──►│ cluster_listen :8005 (CP) └───────────────────────┘ │
│ │
│ Caches: kong_cache / kong_core_cache (mlcache) │
│ LMDB (DB-less/hybrid config) │
└──────────────────────────────────────────────────────┘
│
▼
PostgreSQL (traditional / hybrid CP)
2.2 The request lifecycle (NGINX phases)
Kong installs Lua handlers into NGINX's request phases. The kong/runloop/handler.lua module orchestrates this; the plugins iterator (kong/runloop/plugins_iterator.lua) runs each applicable plugin's phase handler in order:
init / init_worker ─ load config, build router, warm cache, start timers
│
▼ (per request)
certificate ─ pick TLS cert by SNI (kong/runloop/certificate.lua)
rewrite ─ earliest plugin hook (before routing)
access ─ ROUTE the request, then run access plugins:
auth → ACL → rate limiting → request transforms
balancer ─ choose an upstream target; retry on failure
(kong/runloop/balancer/*)
header_filter ─ response header plugins
body_filter ─ response body plugins
log ─ logging/analytics plugins (async)
Each plugin is a handler.lua exposing functions named after these phases (access, header_filter, etc.). The iterator respects plugin priority (higher runs first) and scope (most specific wins).
2.3 Core subsystems
The codebase (see Repository Structure) is organized into clear subsystems. The authoritative module map is the build.modules table in kong-latest.rockspec.
| Subsystem | Path | Responsibility |
|---|---|---|
| Entry / global | kong/init.lua, kong/global.lua |
Process bootstrap, the kong.* global (PDK) wiring. |
| Runloop | kong/runloop/ |
Request handling, router, plugins iterator, balancer, certificate, wasm. |
| Router | kong/router/ |
Match requests to routes (traditional, traditional_compatible, expressions, ATC engine). |
| Config loader | kong/conf_loader/ |
Parse kong.conf, validate, compute derived settings, render NGINX templates. |
| Templates | kong/templates/ |
The NGINX config templates (nginx_kong.lua, nginx.lua, stream variants) and kong_defaults. |
| CLI | kong/cmd/ |
kong start/stop/migrations/config/... commands. |
| DB layer | kong/db/ |
DAO, schema, strategies (Postgres / off), declarative, migrations. |
| Schema | kong/db/schema/ |
Entity schemas, typedefs, metaschema, plugin/vault loaders. |
| PDK | kong/pdk/ |
Plugin Development Kit — the stable API plugins call. |
| Plugins | kong/plugins/ |
The 44 bundled plugins. |
| Vaults | kong/vaults/ + kong/db/dao/vaults.lua |
Secrets backends (OSS ships env). |
| Clustering | kong/clustering/ |
Hybrid mode CP/DP protocol, RPC, config sync, compat. |
| Cluster events | kong/cluster_events/ |
Cross-node cache invalidation events. |
| Cache | kong/cache/, kong/resty/mlcache/ |
Two-layer (LRU + shm) entity caching. |
| DNS | kong/dns/, kong/resty/dns/ |
Resolver (legacy + new global-cache client). |
| Observability | kong/observability/, kong/timing/ |
Tracing, OTLP, request debugging timing. |
| Admin API | kong/api/ |
The RESTful management API (Lapis-based). |
| Status API | kong/status/ |
Read-only health/readiness endpoints. |
| Tools | kong/tools/ |
Shared utilities (string, table, uuid, http, redis, sandbox, queue…). |
2.4 Caching architecture
Kong avoids per-request database hits via two shared-memory caches backed by lua-resty-mlcache (an L1 per-worker LRU + L2 cross-worker shared dict):
kong_cache— general entities (consumers, credentials, tags).kong_core_cache— hot routing entities (services, routes, plugins, upstreams, certificates).
Both are sized by mem_cache_size (default 128m each). Invalidations propagate across nodes via kong_cluster_events. The full shared-dict set (from kong/constants.lua): kong, kong_locks, kong_db_cache, kong_db_cache_miss, kong_cluster_events, kong_healthchecks, kong_rate_limiting_counters, kong_secrets.
3. Deployment Topologies
Choose a topology by setting database and role in kong.conf.
3.1 Traditional (with database)
┌──────┐ ┌──────┐ ┌──────┐
│ Kong │ │ Kong │ │ Kong │ ← all nodes role=traditional, database=postgres
└──┬───┘ └──┬───┘ └──┬───┘
└──────────┼──────────┘
▼
┌─────────────┐
│ PostgreSQL │ ← shared source of truth
└─────────────┘
- Config via Admin API on any node; all nodes share PostgreSQL.
- Changes propagate to other nodes within
db_update_frequency(default 5s). - Simple but every node depends on the database being reachable.
database = postgres
pg_host = 10.0.0.5
pg_database = kong
role = traditional
3.2 DB-less (declarative)
┌──────────────────────┐
│ Kong (database=off) │ ← config loaded from a YAML file into in-memory LMDB
│ declarative_config │
└──────────────────────┘
- No database. Config comes from a declarative file (
declarative_config) orPOST /config. - The Admin API is read-only for entities;
POST /configreplaces the whole config atomically. - Entities live in LMDB; size it with
lmdb_map_size(default2048m). - Best for immutable/GitOps deployments and as the data-plane mode in hybrid.
database = off
declarative_config = /etc/kong/kong.yml
lmdb_map_size = 2048m
3.3 Hybrid (control plane / data plane)
┌──────────────┐ mTLS over cluster_listen :8005 ┌──────────────┐
│ Control Plane│ ─────────────────────────────────► │ Data Plane │ ─► upstreams
│ role=control │ pushes config (deltas/full) │ role=data_ │
│ database=pg │ ◄───────────── heartbeat ────────── │ plane, db=off│
└──────┬───────┘ └──────────────┘
▼
PostgreSQL
- The control plane (CP) holds config in PostgreSQL and exposes the Admin API; it does not proxy traffic.
- Data planes (DP) run DB-less, receive config from the CP over an mTLS channel, and proxy all traffic.
- DPs keep serving even if the CP or database is down (they cache the last good config, optionally persisting it;
declarative_configcan act as a fallback before the first connection). - This is the recommended production topology at scale.
# Control plane
role = control_plane
database = postgres
cluster_listen = 0.0.0.0:8005
cluster_cert = /etc/kong/cluster.crt
cluster_cert_key = /etc/kong/cluster.key
# Data plane
role = data_plane
database = off
cluster_control_plane = cp.internal:8005
cluster_cert = /etc/kong/cluster.crt
cluster_cert_key = /etc/kong/cluster.key
Cluster mTLS modes (cluster_mtls): shared (same cert on all nodes — default), pki (per-DP certs issued by a common cluster_ca_cert), pki_check_cn (also verifies the DP cert's common name against cluster_allowed_common_names).
Generate a shared cert/key with kong hybrid gen_cert.
Important
cluster_dp_labelsonly works with Kong Konnect (SaaS), not self-hosted hybrid. The CP↔DP channel is mTLS-protected end to end; CP connection requests are logged toadmin_access_logunder the server namekong_cluster_listener.
4. System Requirements
| Item | Requirement |
|---|---|
| OS | Linux or macOS (supported_platforms in the rockspec). |
| Runtime | OpenResty with NGINX 1.27.1.2 (bundled in official packages). |
| Lua | LuaJIT (provided by OpenResty). |
| Database | PostgreSQL 9.5+ (traditional / hybrid CP only). DB-less needs none. |
| Memory | Plan for mem_cache_size × 2 (two cache zones) + lmdb_map_size (DB-less/hybrid) + worker overhead. |
| File descriptors | nginx_main_worker_rlimit_nofile defaults to auto (ulimit-bound, capped 16384). |
| Ports | See the port reference. |
4.1 Third-party Lua dependencies
Kong is built on a curated set of lua-resty-* libraries (from kong-latest.rockspec). Operators rarely interact with these directly, but they define Kong's capabilities:
| Dependency | Role |
|---|---|
lua-resty-http |
Outbound HTTP (logging plugins, ACME, AI providers). |
lua-resty-openssl |
TLS/crypto primitives (certs, cluster, JWT). |
lua-resty-acme |
ACME/Let's Encrypt automation. |
lua-resty-healthcheck |
Upstream active/passive health checks. |
lua-resty-session |
Session plugin storage. |
lua-resty-aws / lua-resty-gcp |
Cloud SDKs (Lambda, vaults, AI Bedrock/Gemini). |
lua-resty-ipmatcher |
Fast IP/CIDR matching (ip-restriction, trusted_ips). |
lua-resty-ljsonschema |
JSON Schema validation. |
lua-protobuf / lua-messagepack |
gRPC, clustering RPC. |
kong-lapis |
Web framework powering the Admin API. |
kong-pgmoon |
PostgreSQL driver. |
penlight (pl.*) |
General Lua stdlib (used throughout the CLI). |
lpeg, lua-resty-ada |
Parsing / URL handling. |
5. Configuration Reference
Kong reads a single config file (default /etc/kong/kong.conf) at kong start/kong prepare. The shipped kong.conf.default documents every property with its default. Every property has a KONG_-prefixed, uppercased environment-variable equivalent that overrides the file (e.g. log_level → KONG_LOG_LEVEL). Booleans accept on/off or true/false; lists are comma-separated.
Validate any file with kong check <conf>.
The sections below mirror kong.conf.default.
5.1 General
| Property | Default | Description |
|---|---|---|
prefix |
/usr/local/kong/ |
Working directory (NGINX prefix): temp files, logs, sockets. One per process. |
log_level |
notice |
NGINX error log level. |
proxy_access_log |
logs/access.log |
Proxy access log path (off to disable). |
proxy_error_log |
logs/error.log |
Proxy error log path. |
proxy_stream_access_log |
logs/access.log basic |
TCP/stream access log. |
admin_access_log / admin_error_log |
logs/admin_access.log / logs/error.log |
Admin API logs. |
status_access_log / status_error_log |
off / logs/status_error.log |
Status API logs. |
plugins |
bundled |
Plugins to load (bundled, names, or off). |
vaults |
bundled |
Vault backends to load. |
tracing_instrumentations |
off |
Tracing scope: off, request, all, or specific (db_query, dns_query, router, http_client, balancer, plugin_*). |
tracing_sampling_rate |
0.01 |
Fraction of traces sampled. |
dedicated_config_processing |
on |
Separate worker for CP/DP config handling (DP only). |
anonymous_reports |
on |
Send anonymous usage/error data to Kong. |
proxy_server |
— | Forward proxy (encoded URL) for components configured to use it. |
port_maps |
— | Map external→listen ports (e.g. 80:8000,443:8443) for containerized setups. |
error_template_html/json/xml/plain |
— | Custom error page templates (up to two %s: message, request id). |
5.2 NGINX & listeners
| Property | Default | Description |
|---|---|---|
proxy_listen |
0.0.0.0:8000 reuseport backlog=16384, 0.0.0.0:8443 http2 ssl reuseport backlog=16384 |
Proxy listeners. Suffixes: ssl, http2, proxy_protocol, deferred, bind, reuseport, backlog=N, ipv6only, so_keepalive. off disables proxying. |
stream_listen |
off |
TCP/TLS/UDP stream listeners. |
admin_listen |
127.0.0.1:8001 ..., 127.0.0.1:8444 http2 ssl ... |
Admin API listeners (bound to loopback by default — keep it that way). |
status_listen |
127.0.0.1:8007 reuseport backlog=16384 |
Status API listeners. |
nginx_user |
kong kong |
Worker user/group (falls back to nobody nobody). |
nginx_worker_processes |
auto |
Worker count. |
nginx_daemon |
on |
Daemonize (set off in containers). |
mem_cache_size |
128m |
Size of each of the two entity caches. |
headers |
server_tokens, latency_tokens, X-Kong-Request-Id |
Response headers Kong injects (Server, Via, X-Kong-*-Latency, X-Kong-Upstream-Status, X-Kong-Request-Id, or off). |
headers_upstream |
X-Kong-Request-Id |
Headers injected toward the upstream. |
trusted_ips |
— | CIDR blocks trusted for X-Forwarded-* (sets set_real_ip_from; 0.0.0.0/0,::/0 trusts all). |
real_ip_header |
X-Real-IP |
Header used for the real client IP. |
real_ip_recursive |
off |
Recurse the real-IP chain. |
error_default_type |
text/plain |
Default error MIME when no Accept. |
upstream_keepalive_pool_size |
512 |
Upstream keepalive pool size (per dst ip/port/SNI). |
upstream_keepalive_max_requests |
10000 |
Max requests per upstream keepalive connection. |
upstream_keepalive_idle_timeout |
60 |
Idle timeout (s) for upstream keepalive. |
allow_debug_header |
off |
Enable the Kong-Debug: 1 request header → adds Kong-Route-Id/Kong-Service-* response headers. |
TLS settings
| Property | Default | Description |
|---|---|---|
ssl_cipher_suite |
intermediate |
modern, intermediate, old, fips, or custom. |
ssl_protocols |
TLSv1.2 TLSv1.3 |
Enabled TLS versions (only when custom). |
ssl_cert / ssl_cert_key |
auto-generated | Proxy TLS cert/key (RSA+ECC auto-generated on first start if unset). |
admin_ssl_cert(_key) / status_ssl_cert(_key) |
— | Per-listener TLS material. |
client_ssl / client_ssl_cert(_key) |
off |
mTLS to upstreams (overridable per-Service). |
ssl_session_tickets / ssl_session_timeout |
on / 1d |
TLS session resumption. |
NGINX directive injection
Any property named nginx_<namespace>_<directive> injects <directive> into the corresponding NGINX block. Namespaces: main, events, http, proxy, location, upstream, admin, status, debug, stream, sproxy, supstream, and wasm. Example: nginx_http_ssl_protocols → ssl_protocols in the http {} block; settable via KONG_NGINX_HTTP_SSL_PROTOCOLS. Notable defaults include nginx_http_lua_regex_match_limit = 100000 and nginx_http_lua_regex_cache_max_entries = 8192.
5.3 Datastore (PostgreSQL)
| Property | Default | Description |
|---|---|---|
database |
postgres |
postgres or off (DB-less). |
pg_host / pg_port |
127.0.0.1 / 5432 |
Postgres host/port. |
pg_user / pg_password / pg_database |
kong / — / kong |
Credentials and DB name. |
pg_schema |
— | Schema (respects search_path if unset). |
pg_timeout |
5000 |
Connect/read/write timeout (ms). |
pg_ssl / pg_ssl_verify |
off / off |
TLS to Postgres (verify uses lua_ssl_trusted_certificate). |
pg_max_concurrent_queries |
0 |
Per-worker concurrent-query cap (0 = unlimited). |
pg_semaphore_timeout |
60000 |
Query semaphore timeout (ms). |
pg_keepalive_timeout |
— | Idle pool timeout. |
pg_pool_size / pg_backlog |
— | Per-worker connection pool sizing/backlog. |
pg_ro_* |
inherits | Optional read-only replica connection (pg_ro_host enables it). |
DB-less / declarative
| Property | Default | Description |
|---|---|---|
declarative_config |
— | Path to the declarative YAML/JSON file. |
declarative_config_string |
— | Declarative config inline as a string. |
lmdb_environment_path |
dbless.lmdb |
LMDB directory (under prefix). |
lmdb_map_size |
2048m |
LMDB map size cap (grows on demand). |
5.4 Datastore cache
| Property | Default | Description |
|---|---|---|
db_update_frequency |
5 |
How often (s) to check the datastore for changes. |
db_update_propagation |
0 |
Extra propagation delay (s) for multi-DC replicas. |
db_cache_ttl |
0 |
Entity cache TTL (s); 0 = never expire. |
db_cache_neg_ttl |
— | TTL for cache misses (defaults to db_cache_ttl). |
db_resurrect_ttl |
30 |
How long to serve stale entities when the datastore is unreachable. |
db_cache_warmup_entities |
services |
Entities pre-loaded at startup (also pre-resolves Service DNS). |
5.5 DNS resolver
Legacy resolver: dns_resolver, dns_hostsfile (/etc/hosts), dns_order (LAST,SRV,A,CNAME), dns_valid_ttl, dns_stale_ttl (3600), dns_cache_size (10000), dns_not_found_ttl (30), dns_error_ttl (1), dns_no_sync (off).
New DNS client (new_dns_client = off to enable, exposes stats at /status/dns): resolver_address, resolver_hosts_file, resolver_family (A,SRV), resolver_valid_ttl, resolver_error_ttl (1), resolver_stale_ttl (3600), resolver_lru_cache_size (10000), resolver_mem_cache_size (5m).
5.6 Vaults
| Property | Default | Description |
|---|---|---|
vault_env_prefix |
— | Default prefix for the environment-variable vault. |
The OSS build ships the env vault. Reference secrets in config with {vault://env/<name>}. Additional vault backends (HashiCorp Vault, AWS/GCP/Azure secret managers) are Enterprise.
5.7 Tuning & behavior
| Property | Default | Description |
|---|---|---|
worker_consistency |
eventual |
eventual (async router/balancer rebuild) or strict (deprecated). |
worker_state_update_frequency |
5 |
How often (s) workers check for state changes. |
router_flavor |
traditional_compatible |
traditional_compatible, expressions, or traditional (deprecated). |
lua_max_req_headers / lua_max_resp_headers |
100 / 100 |
Header parse limits (1–1000). |
lua_max_uri_args / lua_max_post_args |
100 / 100 |
Arg parse limits (1–1000). |
5.8 Miscellaneous (Lua/OpenResty)
| Property | Default | Description |
|---|---|---|
lua_ssl_trusted_certificate |
system |
CA bundle for Lua cosockets (and pg_ssl_verify). system auto-detects the distro bundle. |
lua_ssl_verify_depth |
1 |
Cert chain verification depth. |
lua_ssl_protocols |
TLSv1.2 TLSv1.3 |
TLS versions for Lua cosocket connections. |
lua_package_path / lua_package_cpath |
./?.lua;./?/init.lua; / — |
Lua module search paths (for custom plugins). |
lua_socket_pool_size |
256 |
Cosocket pool size per remote. |
untrusted_lua |
sandbox |
Loading of admin-supplied Lua: off, sandbox, or on. |
untrusted_lua_sandbox_requires |
— | Modules allowed via require inside the sandbox. |
untrusted_lua_sandbox_environment |
— | Globals exposed inside the sandbox. |
openresty_path |
— | Explicit OpenResty install to use. |
node_id |
auto | Node UUID (auto-generated if empty). |
5.9 Kong Manager (GUI)
admin_gui_listen (0.0.0.0:8002, 0.0.0.0:8445 ssl), admin_gui_url, admin_gui_path (/), admin_gui_api_url, admin_gui_csp_header (off), admin_gui_ssl_cert(_key), admin_gui_access_log / admin_gui_error_log. Advanced Manager features are Enterprise/Konnect.
5.10 WebAssembly (Wasm)
| Property | Default | Description |
|---|---|---|
wasm |
off |
Enable Proxy-Wasm filter support. |
wasm_filters_path |
— | Directory scanned for .wasm filter modules. |
wasm_filters |
bundled,user |
Which filters are available (bundled, user, names, or off). |
Wasm directives are injected via nginx_wasm_* (and nginx_wasm_shm_kv*, nginx_wasm_wasmtime_*). See the ngx_wasm_module docs for directive details.
5.11 Request debugging
| Property | Default | Description |
|---|---|---|
request_debug |
on |
Enable per-component timing via X-Kong-Request-Debug headers. |
request_debug_token |
random | Token required in X-Kong-Request-Debug-Token (loopback requests are exempt). |
When enabled, send X-Kong-Request-Debug: * (or a comma list of rewrite,access,balancer,response,header_filter,body_filter,log,upstream) to get timing in X-Kong-Request-Debug-Output; add X-Kong-Request-Debug-Log: true to also log it (notice level, prefix [request-debug]). The token is also written to {prefix}/.request_debug_token.
Caution
Deploying Kong behind another proxy can make loopback-exempt debug requests appear "local." Always set a strong
request_debug_tokenin production, or disable request debugging.
6. Operations
6.1 Lifecycle commands
| Action | Command | Signal/behavior |
|---|---|---|
| Start | kong start -c kong.conf |
Spawns NGINX in the prefix. |
| Stop | kong stop |
SIGTERM (immediate). |
| Graceful stop | kong quit |
SIGQUIT; finishes in-flight requests, then SIGTERM after --timeout (default 10s). |
| Reload | kong reload |
SIGHUP; new workers with new config, old workers drain. |
| Restart | kong restart |
stop + start. |
| Health | kong health |
Checks the NGINX PID is running. |
| Prepare prefix | kong prepare |
Renders prefix so you can run nginx -p <prefix> -c nginx.conf directly. |
| Drain | kong drain |
Flips /status/ready to 503 for LB de-registration. |
6.2 Zero-downtime config & version changes
Config-only change (e.g. edit kong.conf):
kong reload -c /etc/kong/kong.conf
Version upgrade (expand/contract migrations):
# 1. Apply backward-compatible migrations while the OLD version keeps running
kong migrations up -c /etc/kong/kong.conf
# 2. Deploy NEW Kong nodes (they run against the partially-migrated schema)
# 3. Once all old nodes are gone, finish the migrations
kong migrations finish -c /etc/kong/kong.conf
kong migrations list/status report the current state with specific exit codes (3 = needs bootstrap, 4 = pending, 5 = new available) — useful in automation.
Graceful node removal:
kong drain # LB stops routing new traffic (/status/ready → 503)
kong quit # finish in-flight requests, then stop
6.3 Backup & restore
- Traditional mode: back up PostgreSQL with
pg_dump. Restore withpg_restore. The schema version is tracked by Kong's migrations. - DB-less mode: your declarative YAML is the backup — keep it in version control.
kong config db_exportproduces a declarative snapshot from a database. - Certificates/keys:
cluster.crt/cluster.key(hybrid) and anyssl_cert/ssl_cert_keyshould be backed up out of band.
6.4 Logs
Logs live under <prefix>/logs/ by default (error.log, access.log, admin_access.log). In containers, set the *_access_log/*_error_log paths to /dev/stdout and /dev/stderr, and nginx_daemon = off. Adjust verbosity with log_level. Dynamic per-node log level can be changed at runtime via the debug Admin API routes (kong/api/routes/debug.lua) without a restart.
6.5 Health & readiness
GET /status(Status API, port 8007) — node metrics + database reachability.GET /status/ready— 200 when the node has loaded config and is ready to proxy; 503 otherwise (and afterkong drain).GET /upstreams/{id}/health(Admin API) — per-target health of a balancer.
Wire /status/ready into your load balancer and Kubernetes readiness probes.
7. Security
Security is the highest-leverage area for a gateway, since it sits on the request path for everything.
7.1 Lock down the Admin API
The Admin API can create, modify, and delete every entity. By default admin_listen binds to loopback (127.0.0.1:8001/8444) — keep it there or restrict it tightly.
Caution
Never set
admin_listento a public interface (e.g.0.0.0.0:8001) without strong protection. If you must expose it, put it behind mTLS, an authenticating proxy, network policy/firewall rules, and ideally a separate management network. In hybrid mode, only the control plane runs the Admin API; data planes have it disabled.
7.2 The untrusted-Lua sandbox
The pre-function/post-function plugins (and some transformation plugins) run admin-supplied Lua. untrusted_lua controls this:
sandbox(default) — restricted environment; no OS access, no globals, only Kong PDK + OpenResty + safe stdlib.off— disables all custom-Lua functionality.on— unrestricted (pre-2.3 behavior; dangerous).
Warning
LuaJIT is not a secure sandbox runtime. The sandbox stops trivial mistakes and casual attackers, not a determined one. The real control is keeping the Admin API private. Allowing extra modules via
untrusted_lua_sandbox_requires(e.g.os,luaposix) can create sandbox escapes — do so only with full understanding.
7.3 Secrets management with Vaults
Avoid plaintext secrets in entity/plugin config by referencing a vault: {vault://env/<name>}. The OSS env vault reads from environment variables (with optional vault_env_prefix). Test resolution with kong vault get env/<name>. Sensitive plugin fields (passwords, signing keys) are the prime candidates.
7.4 Transport security
- Client-facing: enable
sslonproxy_listen; supply realssl_cert/ssl_cert_keyor per-SNI Certificate entities, or automate with theacmeplugin. Tune withssl_cipher_suite(modern/intermediate/old/fips). - To upstreams:
client_ssl+client_ssl_cert(_key)for mTLS, overridable per-Service viaclient_certificate/tls_verify/ca_certificates. - To PostgreSQL:
pg_ssl+pg_ssl_verify(verification useslua_ssl_trusted_certificate). - CP↔DP (hybrid): mTLS over
cluster_listen; choosecluster_mtls=shared/pki/pki_check_cn. Optionally enforce DP cert revocation withcluster_ocsp.
7.5 Request hygiene & forwarded headers
Set trusted_ips to only the proxies in front of Kong so X-Forwarded-* headers from untrusted clients are not honored. Use real_ip_header/real_ip_recursive to derive the true client IP. The ip-restriction, bot-detection, request-size-limiting, and request-termination plugins provide additional perimeter controls.
7.6 Authentication & authorization plugins
Layer auth (key-auth, jwt, oauth2, basic-auth, hmac-auth, ldap-auth, session) with the acl plugin for group-based authorization. A common hardening pattern: enable an auth plugin with an anonymous fallback consumer, then use acl to deny anonymous on protected routes.
7.7 Reporting vulnerabilities
The repository ships a SECURITY.md with the coordinated disclosure process. Report security issues through that channel rather than public issues. The anonymous_reports setting controls Kong's outbound telemetry (to kong-hf.konghq.com); set it off if your policy forbids it.
8. Performance Tuning
8.1 Worker & connection sizing
| Lever | Setting | Guidance |
|---|---|---|
| Worker processes | nginx_worker_processes |
auto = one per CPU core; usually leave it. |
| Worker connections | nginx_events_worker_connections |
auto (ulimit-bound, capped 16384). |
| File descriptors | nginx_main_worker_rlimit_nofile |
auto; raise the OS ulimit for very high concurrency. |
| Listen backlog | backlog=N on *_listen |
Raise with kernel net.core.somaxconn to avoid connection-refused under load. |
8.2 Caching
mem_cache_size(default128mper zone) — increase for large entity counts sodb_cache_warmup_entitiesfits and to reduce LRU churn.db_update_frequency— raising it reduces database load and proxy-latency jitter at the cost of slower propagation.- LRU churning on a too-small cache hurts P99; limiting
pluginsto only those you use reduces churn.
8.3 Upstream keepalive
Tune upstream_keepalive_pool_size (512), upstream_keepalive_max_requests (10000), and upstream_keepalive_idle_timeout (60) to reuse upstream connections and avoid per-request TCP/TLS handshakes.
8.4 Router performance
- Use the default
traditional_compatible(orexpressions) router — both support incremental rebuilds, dramatically shortening rebuild time with many routes. - If you must use
traditionalwith many regex routes, raisenginx_http_lua_regex_cache_max_entriesto at leastregex_paths × 2to avoid CPU spikes. worker_consistency = eventual(default) rebuilds router/balancer asynchronously, preventing long-tail latency during frequent config changes.
8.5 Database
- Use
pg_ro_hostto offload read queries to a replica at scale; setdb_update_propagationto the replication lag. - Tune
pg_pool_size,pg_max_concurrent_queries, andpg_semaphore_timeoutfor connection pressure.
8.6 Known performance considerations
- LuaJIT "NYI" (not-yet-implemented) operations can fall out of JIT compilation and hurt hot paths (community issue #14725). Keep custom Lua in hot phases minimal and JIT-friendly.
- Profile with the
prometheusplugin (latency histograms), request debugging timing, and the perf harness underspec/04-perf/.
9. The Data Model
Kong's runtime behavior is fully described by its core entities (from kong/constants.lua, in dependency order):
workspaces, consumers, certificates, services, routes, snis, upstreams,
targets, plugins, tags, ca_certificates, clustering_data_planes,
parameters, vaults, key_sets, keys, filter_chains
Each entity has a schema under kong/db/schema/entities/. Selected fields:
9.1 Service
| Field | Default | Notes |
|---|---|---|
protocol |
http |
http/https/grpc/grpcs/tcp/tls/udp. |
host, port, path |
— / 80 / — |
Upstream target (or Upstream name). |
retries |
5 |
0–32767. |
connect_/write_/read_timeout |
60000 |
ms. |
tls_verify, tls_verify_depth, ca_certificates, client_certificate |
— | Upstream TLS/mTLS. |
enabled |
true |
Active flag. |
url (shorthand) |
— | Sets protocol+host+port+path in one. |
9.2 Route
| Field | Default | Notes |
|---|---|---|
protocols |
[http,https] |
Mutually-exclusive subsets enforced. |
paths, hosts, methods, headers, snis |
— | Matchers. |
strip_path |
true |
Strip matched prefix upstream. |
preserve_host |
false |
Forward client Host. |
https_redirect_status_code |
426 |
One of 426/301/302/307/308. |
request_buffering/response_buffering |
true |
Body buffering. |
regex_priority |
0 |
Regex tie-breaker. |
path_handling |
v0 |
v0/v1 (v1 deprecated). |
expression, priority |
— | Only in expressions router flavor. |
9.3 Upstream / Target
Upstreams hold the load-balancing algorithm, hash settings, and health-check configuration; Targets are host:port + weight. Pointing a Service's host at an Upstream name activates balancing and health checks (lua-resty-healthcheck).
9.4 Plugins
A Plugin entity has name, a config record (validated by the plugin's schema.lua), enabled, protocols, and optional service/route/consumer foreign keys that determine scope. Some plugins define their own entities (DAOs), e.g. keyauth_credentials, oauth2_tokens, acls.
9.5 Entity validation
Schemas use typedefs (kong/db/schema/typedefs.lua) for reusable field types and entity checks for cross-field rules (e.g. a Service with a stream protocol may not set path; a Route's snis require a TLS protocol). The metaschema (kong/db/schema/metaschema.lua) validates the schemas themselves. JSON-schema validation is available via lua-resty-ljsonschema.
10. Database & Migrations
10.1 The migration system
Migrations live under kong/db/migrations/core/ (numbered 000_base … 025_390_to_3100) plus per-plugin migrations under each kong/plugins/<name>/migrations/. Each migration can define up (backward-compatible) and teardown/finish steps, enabling the expand/contract zero-downtime upgrade model.
| Command | Effect |
|---|---|
kong migrations bootstrap |
Initialize the schema and run all migrations (first install). |
kong migrations up |
Apply new, backward-compatible migrations. |
kong migrations finish |
Apply the remaining (breaking) parts after up. |
kong migrations list |
Show executed/pending/new migrations (exit codes 3/4/5). |
kong migrations status |
JSON status (for automation). |
kong migrations reset |
Erase all data and drop schemas (prompts; --yes to skip). |
Migration locking uses a leader-node mutex; --lock-timeout (default 60s) controls how long other nodes wait.
10.2 Strategies
The DB layer abstracts storage behind strategies (kong/db/strategies/):
postgres— the relational strategy (kong-pgmoondriver), supporting read replicas.off— the DB-less strategy, serving entities from the in-memory LMDB store loaded from declarative config.
The DAO (kong/db/dao/) provides CRUD with validation, pagination, tagging, and cache integration on top of whichever strategy is active.
10.3 Declarative config internals
In DB-less/hybrid, declarative config is parsed (kong/db/declarative/), topologically sorted by entity dependencies (kong/db/schema/topological_sort.lua), marshalled (marshaller.lua), and loaded into LMDB. import.lua/export.lua back the kong config db_import/db_export commands. The hybrid CP serializes config (full snapshot or deltas) and pushes it to DPs over the clustering RPC channel.
11. Repository Structure & Internals
kong/
├── init.lua # process bootstrap (init/init_worker/phases)
├── global.lua # the kong.* global (PDK wiring)
├── constants.lua # bundled plugins, core entities, headers, dicts, protocols
├── meta.lua # version (3.10.0) + dependency pins (nginx)
├── conf_loader/ # parse + validate kong.conf, render NGINX templates
├── templates/ # NGINX config templates (http, stream, kong_defaults, kong_yml)
├── cmd/ # CLI: start/stop/quit/reload/restart/migrations/config/...
│ └── utils/ # prefix_handler, nginx_signals, migrations, process_secrets
├── runloop/ # request handler, plugins_iterator, balancer/, certificate, wasm
│ └── balancer/ # round_robin, consistent_hashing, least_connections, latency, healthcheckers
├── router/ # traditional / traditional_compat / expressions / atc / fields / transform
├── db/ # DAO, schema/, strategies/ (postgres, off), declarative/, migrations/
│ └── schema/entities/ # services, routes, consumers, upstreams, targets, plugins, ...
├── pdk/ # Plugin Development Kit (the stable plugin API)
├── plugins/ # the 44 bundled plugins (handler.lua + schema.lua [+ daos/migrations])
├── vaults/ # secrets backends (OSS: env)
├── clustering/ # hybrid CP/DP: data_plane, control_plane, rpc/, services/sync/, compat/
├── cluster_events/ # cross-node cache invalidation
├── cache/ + resty/mlcache/ # two-layer entity cache
├── dns/ + resty/dns/ # DNS resolvers (new + legacy)
├── observability/ + timing/ # tracing, OTLP, request-debug timing
├── api/ # Admin API (Lapis): endpoints + routes/
├── status/ # Status/readiness API
├── admin_gui/ # Kong Manager serving glue
├── tools/ # shared utils (string, table, uuid, http, redis, sandbox, queue, ...)
└── workspaces/ # workspace scoping
bin/kong # the CLI launcher (resolves cmd, injects NGINX directives, execs `resty`)
kong.conf.default # the complete, commented configuration template
kong-latest.rockspec # authoritative module map + third-party dependency pins
kong-admin-api.yml # OpenAPI spec for the Admin API
autodoc/ # generators for the PDK & CLI reference docs
spec/ , t/ # test suites (busted + Test::Nginx)
changelog/ # per-version changelog fragments
11.1 Entry point flow
bin/kong is a thin resty script: it parses the command, validates it against the known set, injects required NGINX directives (e.g. lmdb_*, lua_ssl_*) into a temporary config, builds the args table, and execs resty -e 'require("kong.cmd.init")(cmd, args)'. kong/cmd/init.lua then dispatches to kong/cmd/<command>.lua. kong/init.lua is the runtime entry that NGINX loads for serving traffic (it wires the phase handlers).
11.2 The PDK (Plugin Development Kit)
kong/pdk/ is the stable, versioned API plugins use, exposed as the kong.* global. Major namespaces:
| PDK namespace | Purpose |
|---|---|
kong.request / kong.response |
Read the client request / build the client response. |
kong.service.request / kong.service.response |
Manipulate the upstream request / read its response. |
kong.client |
Consumer/credential identity, client IP, mTLS. |
kong.router |
Inspect the matched Route/Service. |
kong.ctx |
Per-request and shared context. |
kong.log |
Structured logging + the log serializer. |
kong.cache |
Plugin-accessible entity cache. |
kong.node / kong.cluster |
Node and cluster info. |
kong.vault |
Resolve secret references. |
kong.nginx |
Access NGINX internals. |
kong.tracing / kong.telemetry |
Spans and telemetry. |
kong.plugin |
Plugin metadata helpers. |
Using the PDK (rather than raw ngx.*) is what keeps plugins forward-compatible across Kong versions.
12. Plugin Development
12.1 Anatomy of a Lua plugin
A bundled plugin lives at kong/plugins/<name>/ and consists of:
| File | Required | Purpose |
|---|---|---|
handler.lua |
yes | Phase functions (init_worker, access, header_filter, body_filter, log, …) + a PRIORITY and VERSION. |
schema.lua |
yes | The plugin's config schema (record of fields with defaults/validation). |
daos.lua |
optional | Custom entities the plugin stores (e.g. credentials). |
migrations/ |
optional | DB migrations for those entities. |
api.lua |
optional | Custom Admin API endpoints for the plugin. |
Minimal handler skeleton:
local MyPlugin = {
PRIORITY = 1000, -- higher runs earlier; coordinate with other plugins
VERSION = "1.0.0",
}
function MyPlugin:access(conf)
-- conf is the validated config (per schema.lua)
kong.service.request.set_header("X-Hello", conf.greeting or "world")
end
function MyPlugin:header_filter(conf)
kong.response.set_header("X-Plugin", "my-plugin")
end
return MyPlugin
Minimal schema:
local typedefs = require "kong.db.schema.typedefs"
return {
name = "my-plugin",
fields = {
{ protocols = typedefs.protocols_http },
{ config = {
type = "record",
fields = {
{ greeting = { type = "string", default = "world" } },
},
} },
},
}
12.2 Add a custom plugin in 5 steps
- Create
kong/plugins/my-plugin/handler.luaandschema.lua(above). - Put the plugin on the Lua path: set
lua_package_path(orKONG_LUA_PACKAGE_PATH) to include its parent directory. - Load it:
KONG_PLUGINS=bundled,my-plugin(or addmy-plugintopluginsinkong.conf). - Restart/reload Kong; verify the schema at
GET /schemas/plugins/my-plugin. - Enable an instance:
POST /plugins --data name=my-plugin --data config.greeting=hi.
12.3 Plugin priorities
Priority determines execution order within a phase. As a reference point, authentication plugins run with high priority (so identity is established before authorization/rate-limiting), and logging runs last. When writing a plugin, choose a PRIORITY relative to the bundled plugins it must run before/after.
12.4 Non-Lua plugins
- Go / JavaScript / Python run as external pluginservers: a separate process speaks a protocol to Kong. Configure with
pluginserver_names,pluginserver_<name>_socket,pluginserver_<name>_start_cmd,pluginserver_<name>_query_cmd. Use the matching SDK (go-pdk,kong-pdkfor JS/Python). The runloop integration is inkong/runloop/plugin_servers/. - WebAssembly (Proxy-Wasm): enable
wasm = on, drop.wasmmodules inwasm_filters_path, and attach them via filter chains. The runtime iskong/runloop/wasm.lua.
Tip
The
autodoc/directory contains the generators Kong uses to produce the official PDK and CLI reference documentation from source — useful if you maintain a fork and want matching docs.
13. Observability
| Concern | Mechanism |
|---|---|
| Metrics | prometheus plugin (latency histograms, status codes, bandwidth, upstream health), with a bundled Grafana dashboard (kong/plugins/prometheus/grafana/, Grafana dashboard ID 7424). datadog/statsd plugins for push-based metrics. |
| Tracing | tracing_instrumentations + tracing_sampling_rate in kong.conf; export via the opentelemetry (OTLP) or zipkin plugins. |
| Logging | File/HTTP/TCP/UDP/syslog/loggly plugins; the log serializer (kong.log.serialize) defines the structured log shape. |
| Request timing | Request debugging (request_debug) → X-Kong-Request-Debug-Output; internals under kong/timing/. |
| Health | Status API (/status, /status/ready, /status/dns); upstream health via /upstreams/{id}/health. |
| Node info | GET / on the Admin API; kong version -a. |
Note
Tracing instrumentations are not active in stream (TCP/UDP) mode in the current implementation.
14. Testing Strategy
Kong has an extensive test suite (it is one of the largest parts of the repository):
| Suite | Location | Tool | Scope |
|---|---|---|---|
| Unit | spec/01-unit/ |
busted | Pure-Lua logic, schemas, utilities. |
| Integration | spec/02-integration/ |
busted + helpers | Admin API, proxying, hybrid mode, DB, observability. |
| Plugins | spec/03-plugins/ |
busted | Each bundled plugin end-to-end. |
| Performance | spec/04-perf/ |
custom harness | RPS, flamegraphs (Terraform-driven on cloud instances). |
| Migration | spec/05-migration/ |
busted | Upgrade/migration correctness. |
| PDK / NGINX | t/ |
Test::Nginx (Perl) |
PDK behavior at the NGINX level. |
Test helpers live in spec/helpers/ and spec/internal/. Run with bin/busted (LuaRocks busted). The test harness respects KONG_TEST_* env vars (regular KONG_* vars are ignored during tests for determinism); set KONG_TEST_DONT_CLEAN to inspect the working dir after a run, and KONG_SPEC_TEST_REDIS_HOST to point Redis-dependent tests at a server.
Tip
The mock RPC framework (
spec/helpers/rpc_mock/) lets you intercept and manipulate CP↔DP RPC calls — invaluable when developing hybrid-mode features.
15. Potential Improvements & Tech Debt
Observations drawn from the source and the community issue tracker. These are candidate areas for contributors and operators to watch.
- Router migration (
traditional→expressions):traditionalis deprecated and slated for removal in a future major;path_handling = v1is likewise deprecated. New deployments should standardize ontraditional_compatibleorexpressionsto avoid a forced migration later. - LuaJIT NYI hot paths (#14725): certain operations fall out of JIT compilation and degrade throughput. Hot-phase plugin code should be written JIT-friendly; the project could surface NYI hotspots in the perf harness.
- JWT plugin 500s (#14827): some token shapes surface as HTTP 500 rather than a clean 401/400. Hardening the parser's error mapping would improve operator experience.
- OpenTelemetry gaps: stream-mode tracing is unimplemented; community requests include MCP context propagation via the
_metafield (#14843) and avoiding emitting an emptyhttp.route(#14845). The AI-proxy Gemini driver dropscachedContentduring OpenAI→Gemini transformation (#14837). - AWS Lambda
awsgateway_compatiblepath templates (#14836): path-template reproduction to the resource property is incomplete. - Supply chain / CI: an open issue (#14778) requests restricting
GITHUB_TOKENpermissions in GitHub Actions — a standard supply-chain hardening step worth adopting in forks. - DB import edge case (#12110): declarative DB import can fail without
cert_digestunderca_certificates— validate declarative files withkong config parsebefore importing. - Release cadence concern (#14833): the community has asked about the OSS release cadence; operators relying on the OSS edition should track upstream releases and plan upgrades proactively.
16. Contributing
The repository documents its contribution process in:
CONTRIBUTING.md— how to contribute (issues, PRs, DCO/CLA, style).DEVELOPER.md— setting up a development environment (OpenResty toolchain, dependencies via LuaRocks, running the suites).COMMUNITY_PLEDGE.mdandCODE_OF_CONDUCT.md— community expectations.CHANGELOG.md(+changelog/fragments) — per-version changes; Kong follows SemVer.SECURITY.md— coordinated vulnerability disclosure.
Linting uses luacheck; tests use busted and Test::Nginx. Changelog entries are added as fragment files under changelog/unreleased/ and assembled at release time.
License: Apache 2.0 (Copyright 2016–2026 Kong Inc.).
17. Quick Reference
17.1 Port reference
| Port | Service | Setting |
|---|---|---|
| 8000 / 8443 | Proxy HTTP / HTTPS | proxy_listen |
| 8001 / 8444 | Admin API HTTP / HTTPS | admin_listen |
| 8002 / 8445 | Kong Manager HTTP / HTTPS | admin_gui_listen |
| 8007 | Status API | status_listen |
| 8005 | Cluster (hybrid CP) | cluster_listen |
17.2 Essential commands
kong migrations bootstrap -c kong.conf # first-time DB init
kong start -c kong.conf # start
kong reload # zero-downtime config reload
kong migrations up && kong migrations finish # zero-downtime upgrade
kong drain && kong quit # graceful removal
kong check kong.conf # validate config
kong config db_export kong.yml # snapshot DB to declarative file
kong vault get env/<name> # resolve a secret
kong health # is the node up?
kong version -a # versions of all deps
17.3 Topology cheat sheet
| Goal | database |
role |
|---|---|---|
| Simple shared cluster | postgres |
traditional |
| Immutable / GitOps | off |
traditional |
| Production at scale | postgres (CP) / off (DP) |
control_plane / data_plane |
17.4 Key environment variables
| Env var | Equivalent setting |
|---|---|
KONG_DATABASE |
database |
KONG_PG_HOST / KONG_PG_PASSWORD |
pg_host / pg_password |
KONG_PROXY_LISTEN / KONG_ADMIN_LISTEN |
proxy_listen / admin_listen |
KONG_DECLARATIVE_CONFIG |
declarative_config |
KONG_ROLE |
role |
KONG_LOG_LEVEL |
log_level |
KONG_PLUGINS |
plugins |
KONG_ROUTER_FLAVOR |
router_flavor |
17.5 The 17 core entities
workspaces · consumers · certificates · services · routes · snis · upstreams · targets · plugins · tags · ca_certificates · clustering_data_planes · parameters · vaults · key_sets · keys · filter_chains
17.6 The 44 bundled plugins
Auth: key-auth, basic-auth, jwt, oauth2, hmac-auth, ldap-auth, session Security: acl, cors, ip-restriction, bot-detection, request-size-limiting, request-termination, standard-webhooks Traffic: rate-limiting, response-ratelimiting, proxy-cache, redirect Transform: request-transformer, response-transformer, correlation-id, grpc-gateway, grpc-web Logging: file-log, http-log, tcp-log, udp-log, syslog, loggly Analytics: prometheus, datadog, statsd, opentelemetry, zipkin Serverless: aws-lambda, azure-functions, pre-function, post-function AI: ai-proxy, ai-prompt-decorator, ai-prompt-template, ai-prompt-guard, ai-request-transformer, ai-response-transformer TLS: acme