Project N.O.M.A.D. — Administrator & Developer Guide

This guide is for the people who run, extend, deploy, and audit Project N.O.M.A.D. — system administrators, DevOps, security reviewers, and developers contributing patches. End-user task-flow documentation lives in the User Guide; this document focuses on architecture, full configuration reference, operations, security, and internals.

1. Executive Summary

N.O.M.A.D. is an AdonisJS 6 TypeScript application served behind a Docker Compose stack of seven containers. The admin server speaks Inertia.js to a React 19 SPA, persists structured state in MySQL 8 via Lucid ORM, queues background work in Redis 7 via BullMQ, and orchestrates an arbitrary number of managed services (Kiwix, Ollama, Qdrant, Kolibri, etc.) by talking to the host Docker socket through the dockerode library. Real-time UI updates are pushed to the browser via @adonisjs/transmit (SSE over Redis pub/sub).

Two sidecar containers wrap operational concerns the admin can't safely do itself:

• disk-collector — privileged read-only mount of the host filesystem, publishes JSON disk telemetry every 120 seconds.
• updater — receives signed update requests via a shared volume, edits compose.yml, pulls images, and recreates the admin container (so the admin can update itself without a chicken-and-egg problem).

The product target is a single-tenant LAN appliance, so the design deliberately skips authentication, multi-user accounts, and TLS termination inside the container. SSRF protection is the main exposed-surface defense.

2. Architecture Overview

2.1 Process & container topology

                ┌───────────────────────────────────────────────┐
                │                Host (Debian/Ubuntu)           │
                │                                               │
                │   /var/run/docker.sock                        │
                │            ▲                                  │
                │            │ (bind-mounted)                   │
                │   ┌────────┴────────┐                         │
                │   │ nomad_admin     │ ─ AdonisJS:8080 ─┐      │
                │   │ (this app)      │                  │      │
                │   └─┬───────┬─────┬─┘                  │      │
                │     │       │     │  ┌─────────────────▼────┐ │
                │     │       │     └─►│ nomad_dozzle         │ │
                │     │       │        │ (read-only logs)     │ │
                │     │       │        └──────────────────────┘ │
                │   ┌─▼───┐ ┌─▼───┐                             │
                │   │mysql│ │redis│                             │
                │   └─────┘ └─────┘                             │
                │                                               │
                │   ┌──────────────┐  ┌────────────────────┐    │
                │   │ disk-collec- │  │ updater (sidecar)  │    │
                │   │ tor sidecar  │  │ shared volume IPC  │    │
                │   └──────────────┘  └────────────────────┘    │
                │                                               │
                │   Managed services (created on demand)        │
                │   ┌──────────┐ ┌──────────┐ ┌──────────┐      │
                │   │  kiwix   │ │  ollama  │ │  qdrant  │ …    │
                │   └──────────┘ └──────────┘ └──────────┘      │
                └───────────────────────────────────────────────┘
                          ▲
                          │ HTTP :8080  (Inertia + REST)
                ┌─────────┴─────────┐
                │ Browsers on LAN   │
                └───────────────────┘

Every container shares the project-nomad_default Docker network, named by Docker Compose. Managed services are joined to the same network on creation so the admin can reach them by container name (nomad_kiwix_server, nomad_ollama, …).

Note

The admin container is the only component with access to the Docker socket. Sidecars get scoped capabilities (read-only host mount for disk-collector, write access to compose.yml for updater). Managed services never touch the socket.

2.2 Component inventory

2.3 Technology stack

Note

The package.json in the repo root carries only metadata (name, version, author). All runtime deps live in admin/package.json. The root version field is what semantic-release bumps; the admin one is pinned at 0.0.0.

2.4 Request lifecycle

Browser  ──HTTP──►  nomad_admin :8080
                     │
                     ▼
        Server middleware stack
        ├── ContainerBindingsMiddleware (binds HttpContext)
        ├── @adonisjs/cors (origin '*', credentials true)
        ├── @adonisjs/vite (asset reload in dev)
        ├── @adonisjs/inertia (Inertia handshake)
        ├── @adonisjs/static (public/)
        └── MapsStaticMiddleware (storage/maps/*)
                     │
                     ▼
        Router middleware stack (matched routes only)
        ├── BodyParser (multipart limit 110 MB)
        ├── @adonisjs/shield (XFrame=DENY, CSP/CSRF disabled)
        └── CompressionMiddleware (compression library, gzip)
                     │
                     ▼
        Controller method  ──►  Service layer  ──►  Lucid model / Docker / Ollama
                     │
                     ▼
        Response: Inertia HTML, JSON, SSE stream, or static file

Streaming chat responses bypass the response object's normal flow: the controller flushes SSE headers (Content-Type: text/event-stream, Cache-Control: no-cache, Connection: keep-alive) before any RAG preprocessing so the client connection stays open during the (potentially slow) query rewrite + Qdrant search phase.

2.5 Data flow: AI chat with RAG

1. POST /api/ollama/chat  (validated by chatSchema)
2. OllamaController.chat
   ├── Inject SYSTEM_PROMPTS.default if no system message
   ├── rewriteQueryWithContext()
   │     └─► OllamaService.chat with SYSTEM_PROMPTS.query_rewrite
   │         (skips when <3 user turns or qwen2.5:3b not installed)
   ├── ragService.searchSimilarDocuments(rewritten, 5, 0.3)
   │     └─► Qdrant via @qdrant/js-client-rest
   ├── trim by RAG_CONTEXT_LIMITS based on model size
   ├── splice SYSTEM_PROMPTS.rag_context(joined) before first user msg
   ├── checkModelHasThinking() decides "think" param
   ├── numCtx auto-bump if system prompt > 3k tokens
   ├── chatStream() → for await chunk → response.write SSE frames
   └── on end → chatService.addMessage(assistant, fullContent)

3. System Requirements

Note

The installer requires bash (not sh) and requires sudo — check_is_bash and check_has_sudo are mandatory pre-flight gates. The Debian-family check is also enforced (/etc/debian_version must exist).

4. Configuration Reference

4.1 Environment variables (validated in admin/start/env.ts)

Important

APP_KEY must be ≥ 16 characters or AdonisJS refuses to boot — validation happens during Env.create at startup. The installer generates 32 random alphanumeric chars via tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32.

4.2 Container environment (from install/management_compose.yaml)

4.3 In-app settings (KV store + benchmark settings)

The kv_store table holds free-form, schema-validated settings. Defined keys (from admin/constants/kv_store.ts):

The benchmark_settings table holds keys with their own type system:

4.4 AdonisJS config files (admin/config/*.ts)

4.5 REST + Inertia route map

A consolidated table; see admin/start/routes.ts:1 for source of truth.

Inertia (HTML) routes

JSON API routes (104 endpoints across 9 prefixes)

4.6 Database migrations (in execution order)

naturalSort: true is set in database.ts so the millisecond prefix is respected.

5. Deployment & Scaling

5.1 Reference deployment (single host)

The supplied install/management_compose.yaml is the canonical layout and starts six containers:

Volumes:

volumes:
  /opt/project-nomad/storage  -> /app/storage             # admin
  /var/run/docker.sock        -> /var/run/docker.sock     # admin + dozzle + updater
  /opt/project-nomad/mysql    -> /var/lib/mysql           # mysql
  /opt/project-nomad/redis    -> /data                    # redis
  /opt/project-nomad          -> /opt/project-nomad       # updater (RW)
  /                           -> /host (ro,rslave)        # disk-collector
  nomad-update-shared (named) -> /app/update-shared       # admin <-> updater IPC

extra_hosts: ["host.docker.internal:host-gateway"] is set so the admin can reach services bound to the host (Linux equivalent of the macOS shortcut).

5.2 Scaling considerations

Tip

Run multiple queue workers per host to parallelize background work without running multiple admin instances. The admin process auto-spawns one queue:work --all worker; bring up additional workers as separate compose services pointing at the same compose.yml.

5.3 Updates

Production updates flow through the sidecar pattern:

1. UI button (POST /api/system/update) → SystemUpdateService.requestSystemUpdate
2. Writes {target_tag, ...} JSON to /app/update-shared/update-request
3. nomad_updater polls /shared/update-request once per second
4. On request: sed -i rewrites the admin image tag in compose.yml, docker compose pull, then sequential stop/rm/up -d --no-deps for admin, mysql, redis, dozzle
5. Status JSON is written to /shared/update-status; admin reads it for UI progress

If you've manually customized compose.yml, avoid the in-app update button — sed-based rewrites can leave indentation broken on heavily-edited files. Use update_nomad.sh instead.

Warning

A botched update can wedge compose.yml. The updater always backs up to compose.yml.bak.<timestamp> before editing — keep /opt/project-nomad backed up so you can roll back.

6. Operations

6.1 Day-2 runbooks

Restart the admin only

docker compose -p project-nomad -f /opt/project-nomad/compose.yml restart admin

Tail logs

docker logs -f --tail 200 nomad_admin
docker compose -p project-nomad -f /opt/project-nomad/compose.yml logs -f
# Or use the embedded Dozzle UI at http://<host>:9999

Force-rerun migrations

The admin runs node ace migration:run --force from install/entrypoint.sh on every container start. To re-trigger out-of-band:

docker exec -it nomad_admin node ace migration:run --force

Re-seed the service catalog

docker exec -it nomad_admin node ace db:seed

Idempotent: the seeder uses upserts on service_name.

Back up

sudo /opt/project-nomad/stop_nomad.sh
sudo tar czf nomad-$(date +%F).tar.gz -C /opt project-nomad
sudo /opt/project-nomad/start_nomad.sh

For a faster, app-consistent backup of just the metadata (excluding bulky content), use mysqldump:

docker exec nomad_mysql mysqldump -u nomad_user -p nomad > nomad.sql

Rotate APP_KEY

APP_KEY rotation will invalidate every signed URL in flight (Maps assets, download tokens). Plan a brief outage:

1. Stop the stack.
2. Edit /opt/project-nomad/compose.yml → set new APP_KEY=....
3. Start the stack. Env.create will validate; admin restarts.

Note

Cookies are also encrypted with APP_KEY (@adonisjs/core/helpers Secret), so users may need to re-load their tabs. The stack does not currently use server sessions, so there's no session blow-up to worry about.

6.2 Monitoring & telemetry

• Health: GET /api/health returns {"status":"ok"} plain — wire to Uptime Kuma / Healthchecks.io.
• System info: GET /api/system/info returns disk, RAM, CPU, GPU, and internet-status data structured for the dashboard. Expensive — limit polling to once every 5–15s in custom dashboards.
• Internet status: GET /api/system/internet-status is the lightweight endpoint the UI polls. Override the test URL via INTERNET_STATUS_TEST_URL.
• Container state: read directly from Dozzle (:9999) or docker ps.
• Update status: GET /api/system/update/status returns the JSON the updater sidecar wrote to the shared volume — idle | starting | pulling | pulled | recreating | complete | error.
• Queue: BullMQ stores state in Redis. Use bullmq-cli or any BullMQ dashboard pointed at redis://nomad_redis:6379.

Tip

The admin's pino logger writes to /app/storage/logs/admin.log in production. Mounted to the host at /opt/project-nomad/storage/logs/. Hook Promtail/Vector here to centralize.

6.3 Background job queues

Run dedicated workers via the npm scripts in admin/package.json:

node ace queue:work --queue=downloads
node ace queue:work --queue=model-downloads
node ace queue:work --queue=benchmarks
node ace queue:work --all     # one worker handling every queue

6.4 Pre-deploy checklist

• [ ] Snapshot /opt/project-nomad (or at least mysql, redis, compose.yml)
• [ ] Confirm docker compose version reports v2
• [ ] Verify nvidia-smi works on the host (if GPU-using)
• [ ] Confirm curl http://localhost:8080/api/health returns {"status":"ok"}
• [ ] Set LOG_LEVEL=info (not debug) for production
• [ ] Plan firewall rules for 8080 (admin) and 9999 (Dozzle)
• [ ] Update Crosstalk's GHCR PAT in your CI mirror if you build from source
• [ ] Run a Full benchmark and record the score for hardware regression checks

7. Security

7.1 Threat model

N.O.M.A.D. is built for a trusted LAN appliance:

• No authentication on the admin UI or REST API.
• No HTTPS termination inside the container.
• The admin container has full access to the host Docker socket.
• Sidecar containers have specific elevated privileges (disk-collector reads the host root filesystem; updater writes compose.yml).

This design is acceptable on a private LAN behind a network firewall but unacceptable on the open internet. Do not expose port 8080 publicly.

Caution

Anyone on the LAN can stop services, install/uninstall containers, browse chat history, and trigger updates. If you need network segmentation, put N.O.M.A.D. on a dedicated VLAN.

7.2 SSRF protection

The platform blocks loopback and link-local URLs at the validator layer:

// admin/app/validators/common.ts
const blockedPatterns = [
  /^localhost$/,
  /^127\.\d+\.\d+\.\d+$/,
  /^0\.0\.0\.0$/,
  /^169\.254\.\d+\.\d+$/,        // AWS/GCP/Azure metadata
  /^\[::1\]$/,
  /^\[?fe80:/i,                  // IPv6 link-local
  /^\[::ffff:/i,                 // IPv4-mapped IPv6 (e.g. [::ffff:7f00:1])
  /^\[::\]$/,                    // IPv6 all-zeros
]

Triggered through assertNotPrivateUrl(url) in every download path (maps, ZIM, content-update). RFC1918 ranges (10.x, 172.16-31.x, 192.168.x) are intentionally allowed because LAN mirrors of content are a supported use case.

7.3 Network exposure

Warning

Even with default install, 9999 (Dozzle) is reachable on the LAN. Logs can leak credentials, internal paths, and PII. If your LAN isn't fully trusted, drop the dozzle service from compose.yml.

7.4 Headers and middleware (admin/config/shield.ts)

Important

When you put N.O.M.A.D. behind a reverse proxy (Caddy, Traefik, nginx), the proxy is responsible for HSTS, TLS, and any auth gate. The app trusts X-Forwarded-Proto for generating the maps styles.json URL.

7.5 Secrets handling

• Where: compose.yml contains APP_KEY, DB_PASSWORD, MYSQL_ROOT_PASSWORD, and MYSQL_PASSWORD in plain text.
• Generation: install script uses tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32 for each.
• Hashing: admin/config/hash.ts uses scrypt with cost 16384, blockSize 8, parallelization 1, max memory 32 MiB. Currently unused (no end-user passwords) but available for future auth.
• Benchmark HMAC: BENCHMARK_HMAC_SECRET is a hardcoded shared secret in admin/app/services/benchmark_service.ts. Acceptable — N.O.M.A.D. is open source; the secret only deters casual abuse of the public benchmark endpoint, and a determined attacker can extract it from source.

7.6 Container security posture

Caution

Anyone who can drop a malicious image inside compose.yml and trigger the in-app update can run code as root on the host. Treat write access to /opt/project-nomad/ as equivalent to root.

7.7 Input validation

Vine schemas guard every JSON-accepting endpoint. Patterns to be aware of:

• Filenames: vine.string().trim().minLength(1).maxLength(4096) — the controllers also pass through sanitizeFilename in admin/app/utils/fs.ts.
• URLs: vine.string().url({require_tld: false}) to permit LAN hostnames, paired with assertNotPrivateUrl for SSRF defense.
• Builder-tag format: ^[A-Za-z]+-[A-Za-z]+-\d{4}$ enforced before save.
• Validator schemas for chat, settings, ollama models, ZIM lists, downloads, benchmarks, content updates, and Wikipedia selection. Required keys are documented in §10.

7.8 Dependency hygiene

• npm audit should be clean as of the latest tag — repo carries package-lock.json for reproducible installs.
• Heavy parsers (pdf-parse, tesseract.js, cheerio, fast-xml-parser, jszip, tar) are sandboxed only by Node's process isolation. The admin process is the only thing parsing user-supplied uploads. Untrusted file content is a real attack surface — keep the host updated.

8. Developer Internals

8.1 Repository layout

project-nomad/
├── package.json                 (metadata only)
├── package-lock.json            (root lock — corresponds to bare metadata)
├── Dockerfile                   (admin image, multi-stage, copies into /app)
├── README.md / FAQ.md / CODE_OF_CONDUCT.md / CONTRIBUTING.md
├── .releaserc.json              (semantic-release config)
├── admin/                       AdonisJS application
│   ├── ace.js                   (TypeScript-aware ace launcher)
│   ├── adonisrc.ts              (providers, preloads, command list)
│   ├── package.json             (full dependency set)
│   ├── package-lock.json
│   ├── tsconfig.json / tailwind.config.ts / vite.config.ts / eslint.config.js
│   ├── start/                   (env.ts, kernel.ts, routes.ts)
│   ├── config/                  (13 config files — see §4.4)
│   ├── constants/               (broadcast, kiwix, kv_store, misc, ollama, service_names, zim_extraction)
│   ├── app/
│   │   ├── controllers/         (13)
│   │   ├── exceptions/          (handler.ts + InternalServerErrorException)
│   │   ├── jobs/                (BullMQ workers)
│   │   ├── middleware/          (compression, container_bindings, force_json_response, maps_static)
│   │   ├── models/              (10 Lucid models)
│   │   ├── services/            (17 services — see §8.2)
│   │   ├── utils/               (downloads, fs, misc, version)
│   │   └── validators/          (10 Vine schemas)
│   ├── database/migrations/     (23 migrations + service_seeder.ts)
│   ├── docs/                    (Markdoc files for the in-app docs page)
│   ├── inertia/
│   │   ├── app/app.tsx          (React + Inertia bootstrap)
│   │   ├── pages/               (page-level components)
│   │   ├── components/          (shared UI: chat, maps, file-uploader, markdoc renderers, system info, etc.)
│   │   ├── hooks/               (useDebounce, useSystemInfo, useTheme, …)
│   │   ├── layouts/             (AppLayout, DocsLayout, MapsLayout, SettingsLayout)
│   │   ├── lib/                 (api.ts, classNames, navigation, util, icons, builderTagWords)
│   │   ├── providers/           (ModalProvider, NotificationProvider, ThemeProvider)
│   │   ├── context/             (ModalContext, NotificationContext)
│   │   └── css/app.css
│   ├── public/                  (favicons, branding webp/svg, in-app docs images)
│   ├── resources/views/         (Edge inertia_layout.edge)
│   ├── tests/                   (Japa bootstrap; suites: unit, functional)
│   ├── types/                   (benchmark, chat, collections, docker, downloads, files, kv_store, maps, ollama, rag, services, system, util, zim)
│   └── views/                   (deprecated location of inertia_layout.edge)
├── collections/                 (Markdown todo list + JSON specs for kiwix-categories, maps, wikipedia)
└── install/
    ├── *.sh                     (install/start/stop/uninstall/update + migrate-disk-collector + run_updater_fixes)
    ├── management_compose.yaml  (canonical compose definition)
    ├── sidecar-disk-collector/  (Dockerfile + collect-disk-info.sh)
    ├── sidecar-updater/         (Dockerfile + update-watcher.sh)
    └── wikipedia_en_100_mini_*.zim  (small test ZIMs shipped for offline first-run)

8.2 Service catalog (admin/app/services/)

8.3 Inertia frontend stack

• Bootstrap in admin/inertia/app/app.tsx:
  • Patches crypto.randomUUID for non-HTTPS contexts (LAN deploys often don't terminate TLS at the container).
  • Wraps the app in QueryClientProvider, ThemeProvider, TransmitProvider (real-time SSE), NotificationsProvider, ModalsProvider.
  • Shows React Query Devtools only when environment in ['development', 'staging'].
• Page resolution: resolvePageComponent uses a glob over admin/inertia/pages/**/*.tsx so adding a new page is just dropping a file.
• Shared props: inertia.ts config injects appVersion, environment, and aiAssistantName (cached one minute on the server, invalidated when ai.assistantCustomName is updated).
• Routing: server-side via Adonis routes; the React side is a fully Inertia-style SPA — no React Router, no client-side routing.
• Real-time channels: admin/constants/broadcast.ts defines:

BENCHMARK_PROGRESS:    'benchmark-progress'
OLLAMA_MODEL_DOWNLOAD: 'ollama-model-download'
SERVICE_INSTALLATION:  'service-installation'
SERVICE_UPDATES:       'service-updates'

The frontend subscribes via react-adonis-transmit hooks; the backend publishes via transmit.broadcast(channel, payload).

8.4 Lucid models (admin/app/models/)

All use SnakeCaseNamingStrategy and @column.dateTime({ autoCreate: true, autoUpdate: true }) for timestamps.

8.5 Code patterns and conventions

• Path aliases: admin/package.json imports field maps #controllers/* → ./app/controllers/*.js, #services/*, #middleware/*, #models/*, #validators/*, #jobs/*, #config/*, #start/*, #exceptions/*, #database/*, #tests/*, #providers/*, #policies/*, #abilities/*, #mails/*, #listeners/*, #events/*. Always import via these aliases.
• Hot-reload boundaries (development only): hotHook.boundaries in admin/package.json lists ./app/controllers/**/*.ts and ./app/middleware/*.ts — these get HMR'd via hot-hook.
• DI: services are decorated @inject() and constructor-injected. Controllers take services in the constructor.
• Streaming: never use Adonis's response.send() for SSE; flush headers manually and write to response.response (the underlying Node res).
• Validation first: every controller method that takes JSON should call request.validateUsing(schema) before touching services.
• Errors: throw InternalServerErrorException (admin/app/exceptions/) to surface a 500 with a structured payload; everything else is a plain Error. The custom handler in admin/app/exceptions/handler.ts formats output as JSON when Accept includes application/json.
• Transactions: use Database.transaction() in services; controllers shouldn't open transactions directly.
• Logging: use the injected logger (or import logger from '@adonisjs/core/services/logger') and tag with the class name in square brackets (logger.info('[BenchmarkService] ...')). Pino formatting follows this convention everywhere.

8.6 Testing strategy

• Test runner: Japa 4 with @japa/assert and @japa/plugin-adonisjs.
• Suites in admin/adonisrc.ts:
  • unit — tests/unit/**/*.spec(.ts|.js), 2s timeout
  • functional — tests/functional/**/*.spec(.ts|.js), 30s timeout
• Bootstrap (admin/tests/bootstrap.ts) starts the HTTP server for any suite with name in ['browser', 'functional', 'e2e'].
• Run tests:

cd admin
node ace test                     # all suites
node ace test --suites=unit       # single suite

Note

The test suite is sparse at present — community contributions to expand coverage on services and validators are welcome. CI runs whatever exists on every PR.

8.7 Build pipeline

• Image: Dockerfile in repo root produces the admin GHCR image. Multi- stage: install deps → run node ace build (compiles TypeScript + Vite bundles the React app) → copy minimal runtime layer.
• Local dev: cd admin && npm run dev runs node ace serve --hmr. Vite serves the SPA in middleware mode through @adonisjs/vite/vite_middleware.
• Type-check: npm run typecheck (tsc --noEmit).
• Lint: npm run lint (ESLint flat config plus @tanstack/eslint-plugin-query recommended).
• Format: npm run format (Prettier with @adonisjs/prettier-config).
• Release: semantic-release on main (stable) and rc (prerelease). Each merged commit's conventional-commit type drives version bumps; CI publishes the GHCR image plus a GitHub Release.

8.8 Adding a new managed service

1. Add the entry to admin/database/seeders/service_seeder.ts (or a new migration if you want it added without a re-seed).
2. Add the constant to admin/constants/service_names.ts.
3. If it needs a special install action (volumes, pre-pull steps), wire it in DockerService.createContainerPreflight.
4. Add a Tabler icon name to the icon column so the UI renders it.
5. If it's a UI-driven service (not a dependency), set is_dependency_service = false and provide ui_location.
6. Restart the admin container and verify it appears under Settings → Apps.

8.9 Potential improvements (open opportunities)

• Re-enable CSP (currently csp.enabled: false) once the Vite asset URL pattern is settled.
• Re-enable CSRF protection — would need a token-vending route for the SPA.
• Add structured tracing (OpenTelemetry) — pino is great for log correlation but tracing across BullMQ workers is currently log-only.
• Pluggable embedding model (today the nomic-embed-text choice is hardcoded in RAG service).
• ROCm GPU support (NVIDIA-only today, see setup_nvidia_container_toolkit in install_nomad.sh).
• True multi-tenant mode with auth — would touch routes, controllers, and the (currently disabled) session/auth providers.
• Service install rollback semantics (currently best-effort; failed installs leave a partial container behind).

8.10 Contributing

• File issues at the GitHub tracker; PRs against main for stable, rc for upcoming features.
• Keep PRs scoped — one logical change per PR.
• Run npm run lint && npm run typecheck && node ace test before opening.
• Commit messages follow Conventional Commits (feat:, fix:, chore:, docs:, refactor:, test:) — semantic-release reads them.

9. Troubleshooting

Symptom: Admin container exits at boot with Cannot validate environment variable APP_KEY

Cause: APP_KEY is unset, empty, or shorter than 16 characters.

Solution: Edit compose.yml and ensure the APP_KEY= line under the admin service has a 16+ character value. Restart the stack.

Prevention: Don't hand-edit compose.yml without running through Env.create validation — start the admin and watch logs.

Symptom: Failed to detect hardware: command not found

Cause: systeminformation falls back to OS commands; on a slim base image something like lspci may be missing.

Solution: The Crosstalk-published image already bundles the deps — but if you've built your own image, add apt-get install -y pciutils to the Dockerfile.

Symptom: AI benchmark returns Ollama is not running or not accessible

Cause: Reaching the nomad_ollama service is blocked, the container is down, or the model isn't pulled.

Solution:

docker logs nomad_ollama --tail 50
docker exec nomad_admin curl -s http://nomad_ollama:11434/api/tags

If the second command 404s on /api/tags, Ollama itself is unhealthy. Restart the container; a fresh nomad_ollama will replay the pulled-models volume.

Symptom: Markdoc validation failed

Cause: A .md doc in admin/docs/ uses Markdoc tag syntax that doesn't parse, or the user uploaded a Markdown file with non-Markdoc tags.

Solution: Run node ace doc:validate if you're shipping new in-app docs. For user uploads, the system catches the error and surfaces it through InternalServerErrorException — the Knowledge Base UI shows the failure on the Files tab.

Symptom: Container is unhealthy after pull

Cause: Healthcheck failed within the configured interval × retries window. For nomad_admin that's 30s × 3 = 90s for /api/health to come back 200.

Solution: Tail logs (docker logs --tail 200 nomad_admin); look for env validation errors, MySQL connection refusals, or migration failures. The admin's entrypoint runs migration:run --force then db:seed then queue:work --all & then bin/server.js; failure in any step will show in the logs.

Symptom: Update sidecar reports Failed to update compose.yml image tag

Cause: compose.yml is structurally different from what the sed -i 's|\(image: ghcr\.io/crosstalk-solutions/project-nomad\):.*|\1:${target_tag}|' pattern expects (e.g., you reformatted YAML or pinned a private registry).

Solution: Run update_nomad.sh from the host instead, which uses docker compose pull + up -d --force-recreate and doesn't edit YAML. Restore from a compose.yml.bak.<timestamp> if the sidecar partially edited the file.

Symptom: extractStructuredContent parses only direct body children

Cause: Modern HTML ZIMs nest article content inside multiple wrapper divs; the v1.x extractor only walked direct children. Issue #720.

Solution: Update to a release where this is fixed (track the issue); otherwise use older-format ZIMs (Wikipedia mwoffliner pre-2024).

Symptom: ZIM batch processing stops at 50 articles

Cause: ZIM_BATCH_SIZE = 50 is the per-batch ceiling and hasMoreBatches used the wrong counter, ending iteration early. Issue #719.

Solution: Update; this is fixed in current main. Check admin/constants/zim_extraction.ts for the constant.

Symptom: scanAndSyncStorage queues non-embeddable files

Cause: The sync routine queued every file in kb_uploads regardless of extension, including kiwix-library.xml. Issue #718.

Solution: Update; the embedder now filters by extension list in RagService. Quick fix: move the offending file out of kb_uploads.

Symptom: SSE stream hangs / chat endpoint never sends data

Cause: A reverse proxy is buffering. nginx without proxy_buffering off will hold SSE chunks until they flush.

Solution: Configure your reverse proxy:

location /api/ollama/chat {
  proxy_pass http://nomad_admin:8080;
  proxy_buffering off;
  proxy_http_version 1.1;
  chunked_transfer_encoding off;
}

Caddy and Traefik handle this correctly out of the box.

Warning

Don't put a CDN in front of N.O.M.A.D. — most CDNs buffer SSE and the chat experience will look broken.

10. Quick Reference

10.1 Important file paths

HOST                                            CONTAINER
/opt/project-nomad/compose.yml          ─►      (n/a — host file)
/opt/project-nomad/storage/             ─►      /app/storage
/opt/project-nomad/storage/logs/admin.log       /app/storage/logs/admin.log
/opt/project-nomad/storage/maps/                /app/storage/maps
/opt/project-nomad/storage/zim/                 /app/storage/zim
/opt/project-nomad/storage/kb_uploads/          /app/storage/kb_uploads
/opt/project-nomad/storage/nomad-disk-info.json /app/storage/nomad-disk-info.json (read by admin)
/opt/project-nomad/mysql/                       /var/lib/mysql
/opt/project-nomad/redis/                       /data
named volume: nomad-update-shared               /app/update-shared (admin) + /shared (updater)

10.2 Service names (admin/constants/service_names.ts)

KIWIX:      nomad_kiwix_server
OLLAMA:     nomad_ollama
QDRANT:     nomad_qdrant
CYBERCHEF:  nomad_cyberchef
FLATNOTES:  nomad_flatnotes
KOLIBRI:    nomad_kolibri

10.3 Key constants (admin/constants/)

10.4 Useful one-liners

10.5 Versioning

• App version: package.json (version) — bumped by semantic-release.
• Image tags on GHCR: latest (stable), rc (release candidate), exact vX.Y.Z from each tagged release.
• Migrations: monotonically-numbered; never edit a migration after it's shipped — write a new one.
• DB schema version: implicit (highest applied migration). No schema_version table; AdonisJS Lucid manages this internally.

10.6 Default ports