# stacks Administrator & Developer Guide `cisco-open/stacks` is a small (~30 kLOC of Python source across 24 files at commit `a1fadd8`) Terraform code pre-processor distributed as a Click CLI. This guide is the operator and developer reference: it covers the runtime architecture, every environment-variable and CLI flag, deployment into CI and operator workstations, the security model around inline secrets, and the internals of every shipped module. End-user task documentation — install commands, daily workflows, recipes — lives in the companion User Guide. --- ## 1. Architecture Overview `stacks` is a thin orchestration layer over the `terraform` binary. It is not a daemon, not an agent, and does not maintain its own state — every invocation begins with a directory walk and ends after a `terraform` subprocess returns. All on-disk state is either Terraform-owned (the `.tfstate` files) or transient (`stacks.out/` is rebuilt every run). The runtime is composed of three Python sub-packages: - **`stacks.main`** — the Click entry point. Defines the top-level group `cli`, the seven public commands (`version`, `render`, `diff`, `terraform`, `encrypt`, `decrypt`, plus the `surgery` group), the two hidden commands (`preinit`, `genkey`), and six surgery sub-commands (`edit`, `list`, `import`, `remove`, `move`, `rename`). - **`stacks.cmd`** — command implementations and the `Context` object that encapsulates the resolved layer/stack/env directory tree. Includes `config.py` (env-var defaults), `context.py` (path math), `preinit.py` (file copy and Jinja render), `render.py`, `diff.py`, `terraform.py`, and `surgery.py` (state operations). - **`stacks.helpers`** — reusable utilities. `config.py` (HCL2/JSON/YAML read/write with merging), `crypto.py` (genkey/encrypt/decrypt with PKCS#7 padding and RSA-OAEP-SHA256 + AES-GCM), `directory.py` (`copy_files`, `directory_remove`), `merge.py` (deepmerge wrapper with `_override` semantics), `run.py` (`subprocess.run` wrappers), and `template.py` (Jinja render with `StrictUndefined`). A `stacks.filters` package is referenced from `helpers/template.py` — custom Jinja filters are registered automatically by name from `filters.__all__` and bound to the active `Context`. ### 1.1 Component diagram ```text +---------------------------------------------------------------+ | stacks CLI (Click) | | src/stacks/main.py | | | | version preinit* render diff terraform | | genkey* encrypt decrypt surgery {edit,list, | | import,remove, | | move,rename} | +----+-----------------+----------------+---------------+-------+ | | | | v v v v +----+----+ +-----+------+ +-----+-----+ +-----+-----+ | cmd | | cmd | | helpers | | helpers | | Context |----->| preinit / |-->| template | | crypto | | .tf* | | render / | | (Jinja2, | | (RSA-OAEP | | paths | | diff / | | Strict- | | + AES- | +---------+ | terraform /| | Undefined)| | GCM, | | surgery | +-----------+ | PKCS#7) | +-----+------+ | +-----+-----+ | v | | +-------+-------+ | | | helpers | | | | config (HCL2, |<------+ ENC[...] | | JSON, YAML) | decrypt | | merge | | +-------+-------+ | | v v +-----+----+ +-----+-----+ | helpers | | filesystem | | run | | layer dir + | | (subproc)| | environments| +-----+----+ +------+-----+ | | v v +-----+----+ +-----+-----+ | terraform | | stacks.out| | binary |<----| (rendered) | | (external)| +-----------+ +-----------+ * = hidden command ``` > [!NOTE] > The diagram is rendered from the actual import graph of `src/stacks/` > at commit `a1fadd8`. Every block corresponds to a real `.py` file or > external binary the code invokes. There is no hidden network layer, > daemon, or RPC service. ### 1.2 Component inventory | Component | Path | Role | |---|---|---| | CLI entry point | `src/stacks/main.py` | Click group `cli`, all command and subgroup wiring, stdin handling for `encrypt`. | | Command package | `src/stacks/cmd/__init__.py` | Re-exports `Context`, `diff`, `preinit`, `render`, `surgery`, `terraform`. | | Defaults | `src/stacks/cmd/config.py` | `os.getenv` resolution for the eight `STACKS_*` configuration variables. | | Context | `src/stacks/cmd/context.py` | Single class `Context`. Computes `path`, `root_dir`, `envs_dir`, `stacks_dir`, `stack_dir`, `base_dir`, `work_dir`, `terraform_dir`, `modules_dir`, `universe_file`, `variables_file`, plus `stack`, `layer`, `env`, `subenv`, `instance`. | | Render preparation | `src/stacks/cmd/preinit.py` | File copy + Jinja render + `stacks.tf.json` / `zzz.auto.tfvars.json` generation. | | Render driver | `src/stacks/cmd/render.py` | Calls `preinit`, then optionally `terraform init`. | | Git diff driver | `src/stacks/cmd/diff.py` | `git stash`, double render, `diff -ur --color`. | | Terraform wrapper | `src/stacks/cmd/terraform.py` | `render` + `terraform -chdir= `. | | State surgery | `src/stacks/cmd/surgery.py` | `_list`, `_import`, `remove`, `move`, `edit`. | | Helpers — config IO | `src/stacks/helpers/config.py` | `config_read`, `json_read`, `yaml_read`, `hcl2_read`, `config_write`, `json_write`, `yaml_write`. | | Helpers — crypto | `src/stacks/helpers/crypto.py` | `genkey`, `encrypt`, `decrypt` with recursive list/dict descent. | | Helpers — directory | `src/stacks/helpers/directory.py` | `copy_files`, `directory_remove`. | | Helpers — merge | `src/stacks/helpers/merge.py` | `merge` (deepmerge wrapper with top-level `_override` handling). | | Helpers — run | `src/stacks/helpers/run.py` | `run_command`, `run_script`. | | Helpers — template | `src/stacks/helpers/template.py` | `jinja2_render` with bound filters. | | Filters | `src/stacks/filters/` | Custom Jinja filters: `checksum`, `deepformat`, `encoding`, `filesystem`, `lookup`, `throw`. | ### 1.3 Technology stack | Layer | Technology | |---|---| | Language | Python 3 | | Distribution | `pyproject.toml` (PEP 621) + `uv.lock` for reproducible dev | | CLI | [Click](https://click.palletsprojects.com/) | | Templating | [Jinja2](https://jinja.palletsprojects.com/) with `StrictUndefined` | | Config parsing | [`python-hcl2`](https://pypi.org/project/python-hcl2/), `pyyaml`, stdlib `json` | | Crypto | [`cryptography`](https://cryptography.io/) (RSA-2048, AES-GCM, PKCS#7) | | Merge | [`deepmerge`](https://pypi.org/project/deepmerge/) | | Git | [GitPython](https://gitpython.readthedocs.io/) for `stacks diff` | | External binary | `terraform` | | External binary | `bash` (only for `stacks surgery edit` redirection) | ### 1.4 Render-time data flow The end-to-end pass that `stacks render` makes through the filesystem: 1. **Context construction.** `Context.__init__` infers `path` from the current working directory and `out` from `path/`. It walks four parents up to find `root_dir`. It splits the layer name into env, sub-env, instance. 2. **Output directory reset.** `helpers.directory_remove(work_dir, keep=[".terraform", ".terraformrc", ".terraform.lock.hcl"])`. 3. **File copy.** `helpers.copy_files` is invoked four times: - `stacks_dir → work_dir` (`*.tf`, `*.tfvars.jinja`) prefixed `common_`. - `stack_dir → work_dir` (`*.tfvars.jinja`) prefixed `stack_`. - For each entry under `base_dir`: directories are `shutil.copytree`'d, non-`*.tf` files are `shutil.copyfile`'d, `*.tf` files are then copied via `helpers.copy_files` with prefix `base_`. - `path → work_dir` (`*.tfvars.jinja`) prefixed `layer_`. 4. **First Jinja pass.** `*.tfvars.jinja` files in `work_dir` are rendered in place against a `var` mapping built from on-disk `.tfvars` plus the eight `stacks_*` predefined variables. 5. **Second Jinja pass.** `*.tf` files in `work_dir` are rendered in place against the same `var` mapping (recomputed to include now-rendered `*.tfvars.jinja`). 6. **Variable universe write.** Every variable in the merged set that is not already declared as `variable "..." {}` in the rendered `*.tf` is written into `stacks.tf.json` with an empty body. Terraform sees this as a no-op variable declaration but it lets the universe round-trip. 7. **Variables file write.** All resolved variables go to `zzz.auto.tfvars.json` (the `zzz.` prefix gives it the highest `auto.tfvars` precedence for Terraform). 8. **`terraform init` (optional).** Per `--init` flag. ### 1.5 Variable precedence chain (authoritative) The `preinit` stage merges variables in **ascending priority** order (later layers overwrite earlier layers): ```text 1. environments//env.tfvars 2. environments///*.tfvars (when subenv is present) 3. /*.tfvars (rendered work_dir name: common_*.tfvars.jinja) 4. /*.tfvars (rendered work_dir name: stack_*.tfvars.jinja) 5. /*.tfvars (rendered work_dir name: layer_*.tfvars.jinja) 6. The eight predefined `stacks_*` variables (always last) ``` `helpers.merge` deepmerges dicts and lists at every level, with one special-case rule: top-level keys ending in `_override` are **replaced** rather than merged. > [!IMPORTANT] > Variables from `*.auto.tfvars` (Terraform's own auto-loading) are also > picked up during the universe-file write so that bare resource-only > stacks don't lose pre-existing `.auto.tfvars` declarations. They are not > merged into the Jinja variable set — only Terraform's own variable > resolution sees them. --- ## 2. System Requirements ### 2.1 Operator workstation | Component | Requirement | |---|---| | OS | macOS, Linux, or any platform that runs Python 3 + `bash` + `terraform` | | Python | 3.x (`pyproject.toml` requires Python 3 metadata; the codebase uses `pathlib`, `f-strings`, dict-merge `\|`) | | `terraform` | Any HCL2-compatible release. Version is not pinned by `stacks`. | | `git` | Required for `stacks diff` (uses `gitpython`). | | `bash` | Required for `stacks surgery edit` (`run_script` calls `bash -c`). | | Disk | Each layer materialises into `stacks.out/` — a few hundred KB to a few MB depending on stack size. | ### 2.2 CI runner CI runs of `stacks` need everything an operator workstation needs, plus: - `STACKS_PRIVATE_KEY_PATH` exported (or `STACKS_*` env vars overridden for the runner's filesystem layout). - Network access to Terraform's module registry and provider download endpoints (if the layer pulls remote modules). - Network access to the configured Terraform state backend. ### 2.3 Excluded environments - **No daemon/server.** `stacks` is invoked synchronously and exits. There is no listener and no IPC. - **No native binaries.** All runtime artefacts are Python. - **Windows is not explicitly tested.** The code uses POSIX path conventions (`pathlib` is portable, but `helpers.run.run_script` shells out to `bash`, which is not bundled with Windows). Use WSL. --- ## 3. Configuration Reference ### 3.1 Environment variables (complete) Every environment variable read by `stacks` is enumerated in `src/stacks/cmd/config.py` (with one exception read in `src/stacks/helpers/crypto.py`). There are no others. | Variable | Type | Default | Read in | Purpose | |---|---|---|---|---| | `STACKS_ENVIRONMENTS_DIR` | path | `environments` | `cmd/config.py:4` | Directory below `root_dir` that holds per-environment `.tfvars` files. | | `STACKS_STACKS_DIR` | path | `stacks` | `cmd/config.py:5` | Directory below `root_dir` that holds stack definitions. | | `STACKS_BASE_DIR` | path | `base` | `cmd/config.py:6` | Per-stack base directory containing modules and `.tf` files. | | `STACKS_LAYERS_DIR` | path | `layers` | `cmd/config.py:7` | Per-stack layers directory holding individual layer dirs. | | `STACKS_OUTPUT_DIR` | path | `stacks.out` | `cmd/config.py:8` | Per-layer rendered output directory name. | | `STACKS_TERRAFORM_PATH` | path | `terraform` | `cmd/config.py:9` | Terraform binary `stacks` should invoke. | | `STACKS_EDITOR` | command | unset | `cmd/config.py:10` | Editor for `stacks surgery edit`. | | `EDITOR` | command | `vi` | `cmd/config.py:10` | Fallback editor when `STACKS_EDITOR` is unset. | | `STACKS_PRIVATE_KEY_PATH` | path list | unset | `helpers/crypto.py:87` | Comma-separated list of RSA private key paths used during decrypt. | > [!IMPORTANT] > No defaults exist in `pyproject.toml` for these variables. If you ship > `stacks` to a CI runner, you are responsible for exporting whatever > deviates from the built-in defaults shown above. ### 3.2 CLI flags (complete) Every Click flag and argument that is registered in `main.py`: | Command | Flag / Argument | Type | Default | Required | Description | |---|---|---|---|---|---| | `render` | `--init` | option | `auto` | no | One of `auto`, `always`, `never`. Controls whether `terraform init` runs after preinit. | | `terraform` | `--init` | option | `auto` | no | Same semantics as `render --init`. | | `terraform` | `args` | unprocessed positional | — | no | Forwarded verbatim to the wrapped Terraform binary. | | `genkey` | `--public-key-path` | option | — | yes | Where to write the generated public key. | | `genkey` | `--private-key-path` | option | — | yes | Where to write the generated private key. | | `encrypt` | `--public-key-path` | option | — | yes | RSA public key for encryption. | | `encrypt` | `--from-stdin` | flag | `False` | no | Read plaintext from stdin instead of a positional argument. | | `encrypt` | `string` | positional | — | conditional | Required unless `--from-stdin` is set. | | `decrypt` | `--private-key-path` | option | — | yes | RSA private key for decryption. Comma-separated list accepted. | | `decrypt` | `string` | positional | — | yes | The `ENC[...]` payload, including brackets. | | `surgery import` | `address` | positional | — | yes | Terraform address to import into. | | `surgery import` | `resource` | positional | — | yes | Real-world resource id. | | `surgery remove` | `address` | positional | — | yes | Terraform address to remove from state. | | `surgery move` | `from_address` | positional | — | yes | Source resource address. | | `surgery move` | `to_address` | positional | — | yes | Destination resource address. | | `surgery move` | `to_path` | positional | — | yes | Destination layer path, relative to repo root. | | `surgery rename` | `from_address` | positional | — | yes | Source address (in current layer). | | `surgery rename` | `to_address` | positional | — | yes | Destination address (same layer). | > [!NOTE] > `surgery list`, `surgery edit`, `version`, `preinit`, and `diff` accept > no flags or arguments at all. ### 3.3 Computed paths (Context attributes) Every `*_dir` and `*_file` attribute the rest of the code reads off of `Context`: | Attribute | Computed as | |---|---| | `path` | `cwd` (or constructor argument) | | `root_dir` | `path.parent.parent.parent.parent` (four levels up) | | `envs_dir` | `root_dir / STACKS_ENVIRONMENTS_DIR` | | `stacks_dir` | `root_dir / STACKS_STACKS_DIR` | | `stack_dir` | `path.parent.parent` | | `base_dir` | `stack_dir / STACKS_BASE_DIR` | | `work_dir` | `path / STACKS_OUTPUT_DIR` (or constructor `out`) | | `terraform_dir` | `work_dir / .terraform` | | `modules_dir` | `terraform_dir / modules` | | `universe_file` | `work_dir / stacks.tf.json` | | `variables_file` | `work_dir / zzz.auto.tfvars.json` | | `stack` | `stack_dir.name` | | `layer` | `path.name` | | `env` | `layer.split("_", 1)[0].split("@", 1)[0]` | | `subenv` | `layer.split("_", 1)[0].split("@", 1)[1]` if `@` present | | `env_dir` | `envs_dir / env` | | `subenv_dir` | `env_dir / subenv` if `subenv` else `None` | | `instance` | `layer.split("_", 1)[1]` if `_` present else `None` | The constructor `assert`s that `env_dir.exists()` and (when applicable) `subenv_dir.exists()`. These are the only runtime invariants the code enforces about the directory structure. ### 3.4 Variable precedence — full table Already enumerated in Section 1.5; reproduced here for the operator quick-reference: | Priority | Source | File pattern at read time | Rendered as | |---|---|---|---| | 1 (lowest) | env | `environments//env.tfvars` | n/a | | 2 | sub-env | `environments///*.tfvars` | n/a | | 3 | stacks-wide | `stacks/*.tfvars` | `common_*.tfvars.jinja` | | 4 | stack | `/*.tfvars` | `stack_*.tfvars.jinja` | | 5 | layer | `/*.tfvars` | `layer_*.tfvars.jinja` | | 6 (highest) | predefined | n/a | `stacks_*` keys | --- ## 4. Deployment & Scaling ### 4.1 Workstation deployment ```bash git clone https://github.com/cisco-open/stacks.git cd stacks pip install . ``` Or, in a virtual environment that should track `uv.lock`: ```bash git clone https://github.com/cisco-open/stacks.git cd stacks uv sync ``` Both produce a `stacks` console-script entry point pointing at `stacks.main:cli`. ### 4.2 CI / container deployment A minimal Dockerfile that produces a runnable image: ```dockerfile FROM python:3-slim RUN apt-get update \ && apt-get install -y --no-install-recommends git bash curl unzip \ && rm -rf /var/lib/apt/lists/* # Pin a Terraform version that matches your stacks ARG TF_VERSION=1.x.x RUN curl -fsSL "https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip" \ -o /tmp/tf.zip \ && unzip /tmp/tf.zip -d /usr/local/bin \ && rm /tmp/tf.zip WORKDIR /opt/stacks COPY . . RUN pip install . ENV STACKS_TERRAFORM_PATH=/usr/local/bin/terraform ENTRYPOINT ["stacks"] ``` > [!NOTE] > `stacks` itself does not ship a Dockerfile at commit `a1fadd8`. The > example above is a documented template — adjust pin values to your > environment's standards. ### 4.3 Scaling characteristics `stacks` has no state of its own. There is nothing to scale horizontally; every invocation is independent. Concurrency is constrained by: - **Terraform state locks.** Two simultaneous `stacks terraform apply` runs against the same layer hit the same lock-tfstate. - **Filesystem.** `stacks render` rebuilds `stacks.out/` in place; running two renders for the same layer at the same time will produce a corrupt output directory. - **CI parallelism.** Different layers are independent. Different environments are independent. Different stacks are independent. Run per-layer Terraform plans in parallel without coordination. ### 4.4 Caching strategies - **Skip `terraform init`** — pass `--init never` when you can guarantee the `.terraform/` cache is fresh. - **Persist `.terraform/`** — it is preserved across `stacks render` runs by `directory_remove`'s `keep` list. CI caches that pin `stacks.out/.terraform/` between jobs save the registry round-trip. - **Pre-warm modules** — `terraform init -backend=false` once per job-image build is enough to populate `modules/`. --- ## 5. Operations ### 5.1 Day-to-day runbook - [x] Operator runs `stacks render` to confirm the layer renders cleanly. - [x] Operator runs `stacks diff` to confirm rendered changes match expectations. - [x] Operator runs `stacks terraform plan` to validate Terraform-level effects. - [ ] Operator runs `stacks terraform apply` to execute. - [ ] Operator commits the layer change with the rendered diff in the review. ### 5.2 Key management runbook | Step | Command | Notes | |---|---|---| | Generate | `stacks genkey --public-key-path keys/stacks.pub --private-key-path keys/stacks.key` | One-time per project. Store the private key in a secret manager. | | Encrypt | `stacks encrypt --public-key-path keys/stacks.pub 'value'` | Anyone with the public key can encrypt. | | Decrypt | `stacks decrypt --private-key-path keys/stacks.key 'ENC[...]'` | For ad-hoc verification. Avoid in scripts — `helpers.decrypt` runs automatically during render. | | Rotate | Set `STACKS_PRIVATE_KEY_PATH=new.key,old.key` | Re-encrypt at leisure; drop `old.key` once the round-trip is clean. | > [!TIP] > Treat the private key with the same care as a Terraform state file. > Anyone who can render a layer can decrypt every secret reachable from it. ### 5.3 State surgery runbook | Goal | Command | Underlying Terraform | |---|---|---| | List resources | `stacks surgery list` | `terraform state list` | | Adopt existing resource | `stacks surgery import ` | `terraform import` | | Drop without destroy | `stacks surgery remove ` | `terraform state rm` | | Move within stack | `stacks surgery move ` | `state show` + `import` + `state rm` | | Rename in same layer | `stacks surgery rename ` | Same as move with `to_path = path` | | Inline edit | `stacks surgery edit` | `state pull` + `state push` | > [!NOTE] > `stacks surgery move` re-renders the source layer before pulling the > state; it does **not** automatically re-render the destination layer. > Run `stacks render` in the destination yourself if its template depends > on the just-moved resource. ### 5.4 Health checks There is no health endpoint. Health is verified empirically: ```bash stacks version # CLI is reachable stacks terraform version # wrapped binary is reachable, layer is valid ``` ### 5.5 Logging `stacks` itself emits very little output: - `stacks version` prints two lines. - `stacks encrypt` / `stacks decrypt` print the result. - All other commands either succeed silently or stream the wrapped process's stdout/stderr through. - On unhandled exceptions, Click prints `ClickException.message`; on `subprocess.CalledProcessError`, `helpers.run.run_command` calls `sys.exit(e.returncode)` so the parent shell sees the wrapped binary's return code. > [!TIP] > If you need richer audit logging, wrap `stacks` invocations in your CI > tool's job log. The CLI does not write to disk for logging purposes. --- ## 6. Performance ### 6.1 Hot paths The render pipeline is dominated by: 1. `helpers.copy_files` — one `shutil.copyfile` per matched file. Linear in the number of `*.tf` and `*.tfvars.jinja` files in `stacks_dir`, `stack_dir`, `base_dir`, and the layer. 2. `helpers.jinja2_render` — runs Jinja2 in `StrictUndefined` mode against every `*.tfvars.jinja` and then every `*.tf`. Linear in template size. 3. `helpers.hcl2_read` — invoked many times during preinit (once per variable layer in the precedence chain plus a final pass to enumerate declared variables). HCL2 parsing is the slowest individual step. 4. `terraform init` — when not skipped, dwarfs everything else. ### 6.2 Tuning levers | Lever | How | Effect | |---|---|---| | Skip `init` | `--init never` | Removes the largest single-call cost. | | Cache `.terraform/` | Persist between CI jobs | Cuts `init` to a metadata refresh. | | Reduce template count | Inline rather than split `.tfvars.jinja` files | Fewer `glob` calls + fewer Jinja parses. | | Avoid `_override` keys | Use proper deepmerge semantics | Lets `helpers.merge` skip the special-case branch. | ### 6.3 Resource ceiling For small stacks (the only kind documented in the source — there are no benchmarks shipped), `stacks render` completes in well under a second on modern hardware, dominated by HCL2 parsing. `terraform init` costs are indistinguishable from running `terraform init` directly. > [!NOTE] > `stacks` does not parallelise its file copies, Jinja renders, or HCL2 > reads. The pipeline is sequential by design — variable precedence > requires deterministic merge order. Parallelising at the layer level > (run multiple `stacks` invocations side-by-side) is the supported > scale-out path. --- ## 7. Security ### 7.1 Trust model - **Layer authors are trusted.** A `*.tfvars.jinja` template can call any registered Jinja filter and any predefined variable. The bundled filters include `lookup` and `filesystem` (read access to the repo) and `throw` (which raises). There is no sandbox. - **Operators are trusted.** Anyone who can run `stacks render` against a layer with `STACKS_PRIVATE_KEY_PATH` set has plaintext access to every encrypted value in scope. - **Public keys are not sensitive.** They can be checked into the repository alongside `*.tfvars`. ### 7.2 Secret handling `stacks` defines a single secret-handling envelope: the `ENC[...]` string. ```text ENC[;;;] ``` | Field | Algorithm | Notes | |---|---|---| | `` | RSA-OAEP-SHA256 | The 256-bit AES key, encrypted under the public key. MGF1-SHA256. | | `` | AES-GCM auth tag | 16 bytes, base64. | | `` | random 12-byte IV | `os.urandom(12)`. | | `` | AES-256-GCM ciphertext | Plaintext is PKCS#7-padded to 128-bit boundary first. | **Key generation.** `helpers.crypto.genkey` produces RSA-2048 with public exponent `2**16 + 1`, PEM-encoded, PKCS#8 unencrypted private key, SubjectPublicKeyInfo public key. The private key file has no password — filesystem ACLs are the only access control. **Decryption recursion.** `helpers.crypto.decrypt` walks lists and dicts recursively and decrypts every leaf string that begins with `ENC[` and ends with `]`. Strings without the envelope are returned unchanged. **Multi-key rollover.** `STACKS_PRIVATE_KEY_PATH` accepts a comma-separated list of paths. The decrypt loop tries each key in order, breaking on the first successful RSA-OAEP unwrap. If no key works and `must_decrypt=True` (the default), the last `ValueError` is re-raised. > [!CAUTION] > The private key on disk is **unencrypted PKCS#8**. There is no > passphrase prompt and no integration with OS keychains. Securing the > private key is entirely a filesystem-permissions and secret-manager > concern. ### 7.3 Network surface `stacks` itself opens no sockets. All network activity is delegated to the wrapped `terraform` binary: - Provider plugin downloads (during `init`). - Module registry fetches (during `init`). - State backend reads/writes (during every state-touching command). Operators are responsible for the network policies that apply to those endpoints. ### 7.4 Process surface `stacks` shells out to: | Binary | Invoked from | Args under user control? | |---|---|---| | `terraform` (`STACKS_TERRAFORM_PATH`) | `helpers.run.run_command` via `cmd.terraform`, `cmd.render`, `cmd.surgery` | yes (`stacks terraform `) | | `bash` | `helpers.run.run_script` for `stacks surgery edit` | partially — the script body is interpolated from `STACKS_EDITOR` and the work_dir path | | `git` | via GitPython in `cmd.diff` | no | | `diff`, `cp`, `rm` | via `helpers.run_command` in `cmd.diff` and `cmd.surgery.edit` | no | | Editor (`STACKS_EDITOR` / `EDITOR` / `vi`) | `helpers.run_command` in `cmd.surgery.edit` | yes — the env var is the binary name | > [!WARNING] > `STACKS_EDITOR` and `EDITOR` are passed to `subprocess.run` as the first > element of `argv` — i.e. the binary path. A malicious env var pointing > at an arbitrary executable will run that executable with the temporary > state file as its only argument. Treat these env vars as part of the > trusted operator environment. ### 7.5 Terraform args pass-through `stacks terraform` is decorated with `context_settings={"ignore_unknown_options": True}` and uses `click.UNPROCESSED` for `args`, so the wrapper does no flag validation. Whatever you type is forwarded verbatim (after Click's own argument splitting). This is intentional — it lets every Terraform command and flag work — but it also means there is no defence in depth at the `stacks` layer. ### 7.6 Repo governance - Issue [#23](https://github.com/cisco-open/stacks/issues/23) (open) tracks a Security Policy violation regarding outside collaborators. Operators evaluating the project for adoption should track that issue for resolution. - Issue [#8](https://github.com/cisco-open/stacks/issues/8) (closed) recorded an inquiry about the project's status — see Developer Internals for current code-stewardship signals. --- ## 8. Developer Internals ### 8.1 Repository structure ```text . ├── pyproject.toml # PEP 621 metadata, declares the `stacks` console script ├── uv.lock # uv-pinned dependency lockfile ├── README.md # (high-level overview) ├── CONTRIBUTING.md # contribution policy ├── LICENSE # OSS license ├── docs/ │ ├── 1.1. Considerations before using.md │ ├── 1.2. Stacks vs its alternatives.md │ ├── 2.1. Installation instructions.md │ ├── 2.2. I am starting from scratch.md │ ├── 2.3. I am collaborating to an existing stack.md │ ├── 2.4. I am collaborating to Stacks itself.md │ ├── 3.1.1. Global Terraform code.md │ ├── 3.1.2. Reusable root modules.md │ ├── 3.1.3. Jinja templating for Terraform.md │ ├── 3.1.4. Jinja templating for variables.md │ ├── 3.1.5. Custom Jinja functions.md │ ├── 3.1.6. Inline secret encryption.md │ ├── 3.1.7. Automatic variable initialization.md │ ├── 3.2.1. Terraform state backend configuration.md │ ├── 3.2.2. Terraform provider generation.md │ ├── 3.2.3. Input validation.md │ ├── 3.3.1. stacks render.md │ ├── 3.3.2. stacks terraform.md │ ├── 3.3.3. stacks diff.md │ ├── 3.3.4. stacks encrypt.md │ ├── 3.3.5. stacks decrypt.md │ ├── 3.3.6. stacks surgery list.md │ ├── 3.3.7. stacks surgery import.md │ ├── 3.3.8. stacks surgery remove.md │ ├── 3.3.9. stacks surgery rename.md │ ├── 3.3.10. stacks surgery move.md │ ├── 3.3.11. stacks surgery edit.md │ ├── 3.3.12. stacks version.md │ ├── 3.4. Directory structure.md │ └── 3.5. Special variables.md └── src/stacks/ ├── __init__.py ├── main.py ├── cmd/ │ ├── __init__.py │ ├── config.py │ ├── context.py │ ├── diff.py │ ├── preinit.py │ ├── render.py │ ├── surgery.py │ └── terraform.py ├── helpers/ │ ├── __init__.py │ ├── config.py │ ├── crypto.py │ ├── directory.py │ ├── merge.py │ ├── run.py │ └── template.py └── filters/ ├── __init__.py ├── checksum.py ├── deepformat.py ├── encoding.py ├── filesystem.py ├── lookup.py └── throw.py ``` ### 8.2 Module reference #### `stacks.main` The CLI entry point. Defines `cli`, `version`, `preinit` (hidden), `render`, `diff`, `terraform`, `genkey` (hidden), `encrypt`, `decrypt`, the `surgery` group, and the surgery sub-commands. The only logic the module itself contains is the stdin/positional dispatch in `encrypt`: ```python if from_stdin: string = sys.stdin.read() if not string: raise click.ClickException("No input received from stdin.") elif string is None: raise click.ClickException("You must provide either a string argument or use --from-stdin.") ``` Everything else is delegated to `cmd.*` and `helpers.*`. #### `stacks.cmd.config` A single file of `os.getenv` calls. Read once at import, the eight constants (`ENVIRONMENTS_DIR`, `STACKS_DIR`, `BASE_DIR`, `LAYERS_DIR`, `OUTPUT_DIR`, `TERRAFORM_PATH`, `EDITOR`, plus the implicit `EDITOR` fallback) are imported by `cmd.context`, `cmd.render`, `cmd.surgery`, and `cmd.terraform`. To override a value, export the corresponding `STACKS_*` env var **before** the Python process starts — the constants are not re-read at command time. #### `stacks.cmd.context` Defines `Context`. The constructor's first conditional handles the re-entrant case where the `out` directory passed in has the same name as `STACKS_OUTPUT_DIR` and shares a parent with `path` — used by `cmd.diff` to render against alternative output directories (`stacks.out.old`, `stacks.out.new`). `Context` does not cache anything across calls. Re-entrancy is achieved by simply constructing a new `Context` per command. #### `stacks.cmd.preinit` The 100-line workhorse. Reads: - `helpers.directory_remove` to wipe `work_dir`. - `helpers.copy_files` four times. - `shutil.copytree` / `shutil.copyfile` for non-`*.tf` items in `base_dir`. - `helpers.hcl2_read` against the variable-precedence chain to build the Jinja `var` mapping. - `helpers.jinja2_render` twice (once for `*.tfvars.jinja`, once for `*.tf`). - `helpers.json_write` for `stacks.tf.json` and `zzz.auto.tfvars.json`. The `variables_predefined` dict is the source of the eight `stacks_*` Jinja variables. Note the `# TODO: replace 'env.tfvars' with '*.tfvars' after all stacks have been upgraded to v2` comment in the `stacks_environments` map — a v2 migration is in flight. #### `stacks.cmd.render` Three lines of substance. Calls `preinit.preinit`, then conditionally runs `terraform -chdir= init` based on the `--init` flag. #### `stacks.cmd.diff` Drives the rendered-vs-rendered comparison: 1. Stash with `gitpython`. 2. Render the stashed working tree into `.old`. 3. `git stash pop`. 4. Render the live tree into `.new`. 5. `diff -ur --color` between them. 6. `rm -rf` both. Uses `helpers.run_command` for `diff` and `rm` so the user sees coloured output streamed to the terminal. #### `stacks.cmd.terraform` Seven lines. Calls `render.render(ctx, init)` then `helpers.run_command(config.TERRAFORM_PATH, f"-chdir={ctx.work_dir}", *args)`. Returns the completed `subprocess.CompletedProcess`. #### `stacks.cmd.surgery` Implements `_list`, `_import`, `remove`, `move`, `edit`. The non-trivial logic is in `move` and `edit`: - `move` parses the source resource id from `terraform state show -no-color `, captures the first attribute with `id` from the parsed HCL2 output, then chains `_import` against a fresh `Context(path=to_path)` followed by `remove` against the original. The author's comment is preserved verbatim in source: *"this is ugly but it works, do NOT touch."* - `edit` runs render, `terraform state pull > old.tfstate`, copies that to `new.tfstate`, opens `new.tfstate` in the editor, runs `diff -ru --color` (with `|| true` because `diff` returns non-zero on any difference), and prompts `Proceed with changes? [y/N]` before `terraform state push new.tfstate`. Surgery sub-commands like `list` are exposed in `main.py` under `stacks.cmd.surgery._list` because Python reserves `list` as a builtin — the underscore-prefixed names are intentional. #### `stacks.helpers.config` Generic decoder dispatch: - `config_read(patterns, should_decrypt, must_decrypt, decoderfunc, **decoderargs)` globs every pattern, reads each matching file with `decoderfunc`, and successively `merge`s them into `data`. Calls `decrypt` on the result when `should_decrypt` is `True`. - `json_read`, `yaml_read`, `hcl2_read` are thin wrappers passing `json.load`, `yaml.safe_load`, `hcl2.load` respectively. All default `should_decrypt=True, must_decrypt=True`. - `config_write` is the dual; `json_write` and `yaml_write` are wrappers with sensible indenting (2-space JSON, 2-space YAML with `width=1000`). #### `stacks.helpers.crypto` Self-contained envelope crypto. The encrypt path: 1. Generate 32 random bytes for the AES-256 key. 2. Generate 12 random bytes for the GCM IV. 3. PKCS#7-pad plaintext to 128-bit boundary. 4. AES-GCM encrypt. 5. RSA-OAEP-SHA256 encrypt the AES key under the public key. 6. Concatenate base64 fields with `;` separators, wrap in `ENC[...]`. The decrypt path is the reverse, plus a list/dict recursion guard: ```python elif isinstance(data, list): return [decrypt(...) for item in data] elif isinstance(data, dict): return {key: decrypt(...) for key, value in data.items()} ``` Strings without the `ENC[` envelope are passed through verbatim. #### `stacks.helpers.directory` Two functions. - `copy_files(src, dst, include=None, prefix="")` — walks `src`, shutil-copies any file whose name matches one of the `include` fnmatch patterns, into `dst/`. Creates `dst` if missing. - `directory_remove(path, keep=[])` — iterates `path` and removes every child whose name is not in `keep`. Used to wipe `stacks.out/` while preserving `.terraform/`, `.terraformrc`, `.terraform.lock.hcl`. #### `stacks.helpers.merge` A four-line function around `deepmerge.always_merger.merge` with one custom rule: top-level keys in `b` ending in `_override` replace the corresponding key in `a` rather than merging. The TODO in source notes this is a layering hack pending HCL-language support for override semantics. #### `stacks.helpers.run` `run_command(*argv, interactive=True)` is the only subprocess wrapper. When `interactive=True` (the default), stdout/stderr stream to the terminal and a non-zero exit calls `sys.exit(e.returncode)`. When `interactive=False`, output is captured and `CalledProcessError` is re-raised — used by `cmd.surgery.move` to parse `terraform state show` output. `run_script(script)` shells out to `bash -c