# cheat.sh Admin Guide

A complete, source-based reference for operators, developers, and architects who
want to run, extend, or understand **cheat.sh** — the community-driven cheat sheet
aggregation service.

This guide covers the system architecture, every configuration parameter, the
deployment model, security posture, operational procedures, and the internals
needed to modify or self-host the service. For end-user query syntax, see the
**cheat.sh User Guide**.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Architecture](#2-architecture)
3. [Request Lifecycle](#3-request-lifecycle)
4. [The Adapter System](#4-the-adapter-system)
5. [Routing Engine](#5-routing-engine)
6. [System Requirements](#6-system-requirements)
7. [Configuration Reference](#7-configuration-reference)
8. [Deployment](#8-deployment)
9. [Operations](#9-operations)
10. [Caching](#10-caching)
11. [Security](#11-security)
12. [Repository Structure](#12-repository-structure)
13. [Module Reference](#13-module-reference)
14. [Code Patterns & Conventions](#14-code-patterns--conventions)
15. [Testing](#15-testing)
16. [Troubleshooting (Operator)](#16-troubleshooting-operator)
17. [Potential Improvements](#17-potential-improvements)
18. [Quick Reference](#18-quick-reference)

---

## 1. Executive Summary

cheat.sh is a Python web service that presents a **single curl/browser/editor
interface** to many community cheat sheet repositories and StackOverflow. Its core
design idea is that **the URL path is the query** and the response is plain,
pasteable text (or, for browsers, an HTML rendering of that same text).

**Key technical characteristics:**

- **Stateless HTTP service** built on Flask, served in production by a gevent
  WSGI server for high concurrency.
- **Pluggable adapter architecture** — each upstream source (tldr-pages, the
  `cheat` project, Learn X in Y Minutes, RosettaCode, StackOverflow, plus computed
  sources like RFC/OEIS/chmod) is a subclass of a common `Adapter` base.
- **Redis-backed caching** keyed by `topic_type:topic`, with adapter-level control
  over what is cacheable.
- **Content-negotiated rendering** — terminal user agents get ANSI text, browsers
  get HTML produced by piping the ANSI output through an `ansi2html` shell filter.
- **Self-hostable** via Docker Compose (app container + Redis), with all cheat
  sheets fetched at image-build time.

> [!NOTE]
> There is no database, no user accounts, and no authentication. State is limited
> to the Redis cache and optional per-client "last query" / session-id cookies.
> This keeps the service simple and horizontally scalable.

---

## 2. Architecture

### High-level topology

```text
                         ┌──────────────────────────────────────────┐
   curl / wget / xh ───► │  gevent WSGIServer  (bin/srv.py)          │
   browser          ───► │  ── wraps ──                              │
                         │  Flask app          (bin/app.py)          │
                         │    • User-Agent → ansi | html             │
                         │    • rate limiting (lib/limits.py)        │
                         │    • POST = cheat-sheet submission        │
                         └───────────────┬──────────────────────────┘
                                         │ cheat_wrapper(query, opts, fmt)
                                         ▼
                         ┌──────────────────────────────────────────┐
                         │  cheat_wrapper  (lib/cheat_wrapper.py)    │
                         │    parse query → route → postprocess      │
                         └───────────────┬──────────────────────────┘
                                         │ get_answers(topic)
                                         ▼
                         ┌──────────────────────────────────────────┐
                         │  Router  (lib/routing.py)                 │
                         │    routing table → topic_type(s)          │
                         │    cache lookup (lib/cache.py → Redis)    │
                         └───────────────┬──────────────────────────┘
                                         │ adapter.get_page_dict(topic)
                                         ▼
         ┌───────────────────────────────────────────────────────────────────┐
         │  Adapters  (lib/adapter/*)                                         │
         │  cheat.sheets · cheat · tldr · learnxiny · rosetta · question      │
         │  late.nz · fosdem · translation · rfc · oeis · chmod · internal    │
         └───────────────┬───────────────────────────────────────────────────┘
                         │ reads local git checkouts (upstream/) or runs commands
                         ▼
         ┌───────────────────────────────────────────────────────────────────┐
         │  Local cheat-sheet repositories (fetched by lib/fetch.py)         │
         │  + external upstream binary for StackOverflow questions           │
         └───────────────────────────────────────────────────────────────────┘
                                         │
                         ┌───────────────▼──────────────────────────┐
                         │  Frontends  (lib/frontend/*)              │
                         │   ansi.py  → Pygments-highlighted ANSI    │
                         │   html.py  → ansi2html.sh → HTML page      │
                         └──────────────────────────────────────────┘
```

### Component inventory

| Component | File(s) | Responsibility |
|-----------|---------|----------------|
| Production server | `bin/srv.py` | gevent WSGI server, monkey-patches stdlib, binds host:port |
| Web application | `bin/app.py` | Flask routes, User-Agent detection, rate limiting, POST handling, logging |
| Wrapper | `lib/cheat_wrapper.py` | Parse/normalize the query, dispatch to router and search, postprocess |
| Router | `lib/routing.py` | Map a topic to one or more adapters via the routing table; manage cache |
| Adapters | `lib/adapter/*.py` | Retrieve answers from each source |
| Frontends | `lib/frontend/*.py` | Render answer data as ANSI or HTML |
| Formatting | `lib/fmt/*.py` | Comment beautification, internal-page colorization, markdown |
| Coloring lib | `lib/panela/*.py` | Internal ANSI/color manipulation library (`panela`) |
| Config | `lib/config.py` | Layered configuration (defaults → files → env) |
| Cache | `lib/cache.py` | Redis (or no-op) cache |
| Rate limiting | `lib/limits.py` | Per-IP query caps |
| Fetch/update | `lib/fetch.py` | Clone and update upstream cheat-sheet repos; invalidate cache |
| Standalone | `lib/standalone.py` | Offline command-line entry point (no server, no cache) |
| Client | `share/cht.sh.txt` | The `cht.sh` shell client served at `/:cht.sh` |

> [!NOTE]
> The service code lives in `lib/` and `bin/`. The `share/` directory holds the
> client script, shell completions, editor helpers, static assets, and the
> `ansi2html.sh` converter used for HTML rendering.

---

## 3. Request Lifecycle

A `GET /<topic>` request flows through the following steps (`bin/app.py:answer`):

1. **User-Agent inspection.** `_is_html_needed()` checks the User-Agent against
   `PLAIN_TEXT_AGENTS` (`curl`, `wget`, `httpie`, `lwp-request`, `python-requests`,
   `openbsd ftp`, `powershell`, `fetch`, `aiohttp`, `xh`). A match means ANSI
   output; anything else (a browser) means HTML.
2. **Option parsing.** `options.parse_args(request.args)` turns query-string flags
   into an options dict (see [§7](#7-configuration-reference) → request options).
3. **Special early exits.** Favicon/apple-touch icons return empty; `:last` returns
   the caller's previous query (requires a session-id cookie); POST requests are
   routed to `process_post_request()` (cheat-sheet submission).
4. **Language negotiation.** `get_answer_language()` derives a human language from
   the `*.cheat.sh` subdomain, a `?lang=` argument, or the `Accept-Language`
   header.
5. **Rate limiting.** If the topic contains `+` (a multi-word query),
   `Limits.check_ip()` is consulted; over-limit requests get `429`.
6. **Dispatch.** `cheat_wrapper(topic, request_options, output_format)` runs the
   query and returns `(result, found)`.
7. **Rendering & logging.** The result is returned as `text/plain` (ANSI clients)
   or HTML (browsers). Every query is appended to the query log.

Inside `cheat_wrapper()` (`lib/cheat_wrapper.py`):

1. **Sanitize** the query (strip `< > "`).
2. **Add section name** — turn the first space/`+` into a `/` so `python lambda`
   becomes `python/lambda` (with care for `g++` and friends).
3. **Rewrite aliases & editor section names** (`emacs:go-mode` → `go`,
   `c++` → `cpp`).
4. **Parse** for a `~keyword` search and search options.
5. **Resolve** — `find_answers_by_keyword()` for searches, otherwise
   `get_answers()` (the router).
6. **Postprocess** each answer (add language-appropriate comments, filter by
   keyword).
7. **Visualize** via the chosen frontend (`ansi`, `html`, or `json`).

> [!IMPORTANT]
> The `json` output format exists in `cheat_wrapper()` but `bin/app.py` only ever
> requests `html` or `ansi`. **There is no JSON HTTP endpoint.** Treat answers as
> text or HTML only.

---

## 4. The Adapter System

Every source of answers is an **adapter** — a subclass of `Adapter`
(`lib/adapter/adapter.py`). This is the central extensibility mechanism.

### Base class contract

| Method | Purpose |
|--------|---------|
| `_get_list(prefix=None)` | Return the list of topics this adapter serves |
| `_get_page(topic, request_options)` | Return the answer text (or dict) for a topic |
| `is_found(topic)` | Whether this adapter has the topic (root only, by default) |
| `is_cache_needed()` | Whether answers should be cached (`_cache_needed`) |
| `get_page_dict(topic, ...)` | Wraps `_get_page` into the standard answer dict |
| `local_repository_location()` | Where the adapter's git checkout lives |
| `fetch_command()` / `update_command()` | Shell commands to clone/update the source |

Adapter discovery is automatic: `all_adapters()` walks `Adapter.__subclasses__()`
recursively, so **adding a new adapter is as simple as defining a subclass** in
`lib/adapter/` and adding its name to `adapters.active` in config.

### Adapter taxonomy

```text
Adapter (abstract)
├── RepositoryAdapter            file-backed cheat sheets
│   └── GitRepositoryAdapter     adds git clone/pull/state for cache invalidation
│       ├── CheatSheets          chubin/cheat.sheets        (native sheets)
│       │   └── CheatSheetsDir   directory listings (LANG/)
│       ├── Cheat                cheat/cheatsheets
│       ├── Tldr                 tldr-pages/tldr            (.md, multi-platform)
│       ├── LearnXinY            adambard/learnxinyminutes  (:learn pages)
│       ├── Rosetta              acmeism/RosettaCodeData
│       └── Latenz               chubin/late.nz            (curlable latencies)
├── CommandAdapter               answer = stdout of a shell command
│   ├── Fosdem                   current FOSDEM slide (demo)
│   ├── Translation              external get_translation binary
│   ├── AdapterRfc               share/adapters/rfc.sh
│   ├── AdapterOeis              share/adapters/oeis.sh
│   └── AdapterChmod             share/adapters/chmod.sh
├── UpstreamAdapter              proxy to another cheat.sh server
│   └── Question                 StackOverflow via external upstream binary
├── InternalPages                :help, :intro, :list, :styles, etc.
│   └── UnknownPages             fuzzy "did you mean…" suggestions
└── Search                       marker adapter for the search route
```

### Notable adapters

- **`CheatSheets`** — the native sheets from `chubin/cheat.sheets`, hierarchically
  organized; directory names are stored with a leading `_` that is stripped for
  queries.
- **`Tldr`** — reads tldr-pages `.md` files, searching `common`, `linux`, `osx`,
  `sunos`, `windows`, `android` in order, and applies a trivial Markdown
  transform.
- **`LearnXinY`** — parses each language's `*.html.markdown` file from Learn X in Y
  Minutes into blocks; powers the `:learn` pages. Each language has its own
  subclass with block-separator regexes.
- **`Question`** — answers free-form programming questions. It shells out to an
  external binary at `path.internal.bin.upstream`; if that binary is absent, it
  **falls back to `UpstreamAdapter`**, proxying to `upstream.url`
  (default `https://cheat.sh`). It also runs language detection (`polyglot`) to
  support non-English questions.
- **`UpstreamAdapter`** — proxies to another cheat.sh instance; on connection
  failure it returns the "Are you offline?" ASCII banner and marks the answer
  non-cacheable.

> [!TIP]
> When self-hosting without the proprietary StackOverflow upstream binary, the
> `question` adapter will transparently proxy to the public `https://cheat.sh`.
> Set `upstream.url` to your own mirror, or accept the public default.

---

## 5. Routing Engine

`Router` (`lib/routing.py`) decides which adapter(s) answer a topic, using a
**routing table** assembled from three configurable lists plus a default:

```text
routing_table = routing.pre + routing.main + routing.post
```

Each entry is `(regexp, adapter_name)`. `get_topic_type()` walks the table; every
regex that matches **and** whose adapter reports `is_found(topic)` contributes a
route. If more than one matches, the trailing (default) route is dropped. If none
match, `routing.default` (`question`) is used.

### Default routing table (from `lib/config.py`)

**`routing.pre`** (highest priority):

| Regex | Adapter | Matches |
|-------|---------|---------|
| `^$` | `search` | Empty query |
| `^[^/]*/rosetta(/\|$)` | `rosetta` | `…/rosetta` paths |
| `^rfc/` | `rfc` | `rfc/NNN` |
| `^oeis/` | `oeis` | `oeis/ANNN` |
| `^chmod/` | `chmod` | `chmod/NNN` |
| `^:` | `internal` | Special pages (`:help`, …) |
| `/:list$` | `internal` | Any `…/:list` |
| `/$` | `cheat.sheets dir` | Trailing slash → directory listing |

**`routing.main`** (the cheat-sheet sources, tried in order):
`cheat.sheets`, `cheat`, `tldr`, `late.nz`, `fosdem`, `learnxiny`.

**`routing.post`**:

| Regex | Adapter | Matches |
|-------|---------|---------|
| `^[^/ +]*$` | `unknown` | A single bare word with no match → "did you mean" |
| `^[a-z][a-z]-[a-z][a-z]$` | `translation` | Locale pairs like `en-de` |

**`routing.default`**: `question` (StackOverflow).

### Special routing behaviors

- **`:random`** — `handle_if_random_request()` rewrites `/LANG/:random` to a random
  real topic under that prefix before routing.
- **Explicit type** — a topic of the form `type:topic` (e.g. `tldr:tar`) forces a
  single adapter.
- **`question` caching** — question answers use a legacy `q:<topic>` cache key and
  are reused even across cache-format changes, because they are expensive to
  generate.

> [!NOTE]
> The set of live adapters is `adapters.active ∪ adapters.mandatory`. To disable a
> source, remove it from `adapters.active`; to add one, add its name there (and
> ship the adapter class).

---

## 6. System Requirements

### Runtime

| Requirement | Detail |
|-------------|--------|
| Python | 3.x (the code carries some Python-2 compatibility shims but targets 3) |
| Redis | For caching (`cache.type: redis`); optional if `cache.type: none` |
| `git` | To fetch/update upstream cheat-sheet repositories |
| `bash` + `gawk` + `sed` | Required by `ansi2html.sh` for HTML rendering |
| `rlwrap` | Only on clients, for `cht.sh --shell` |

### Python dependencies (`requirements.txt`)

| Package | Role |
|---------|------|
| `gevent` | Async WSGI server (production) |
| `flask` | Web framework / routing |
| `requests` | Upstream proxying (Question/Upstream adapters) |
| `pygments` | Syntax highlighting + the list of color styles |
| `redis` | Cache backend |
| `fuzzywuzzy` / (optional `rapidfuzz`) | "Did you mean" fuzzy topic matching |
| `langdetect`, `polyglot`, `PyICU`, `pycld2` | Human-language detection for questions |
| `colored`, `colorama` | Terminal color handling |
| `pyyaml` | Config + RosettaCode metadata parsing |
| `dateutils`, `cffi`, `python-Levenshtein` | Supporting libraries |
| `wheel`, `black`, `pytest` | Build / format / test tooling |

> [!WARNING]
> `polyglot` + `PyICU` + `pycld2` have native build dependencies (ICU, a C++
> toolchain). The Dockerfile installs these via Alpine packages and a temporary
> build toolchain; expect non-trivial build steps if installing on bare metal.

---

## 7. Configuration Reference

Configuration is centralized in `lib/config.py` as a single `CONFIG` dictionary
with **dotted keys**. Paths are made absolute relative to `path.workdir`.

### Resolution order (highest wins)

1. **Environment variables** (`CHEATSH_*`)
2. **Config file in the work dir** — `$WORKDIR/etc/config.yaml`
   (or `$CHEATSH_CONFIG`)
3. **Config file in the project dir** — `<project>/etc/config.yaml`
4. **Built-in defaults** in `_CONFIG`

The work dir defaults to `$HOME/.cheat.sh` and can be overridden with
`CHEATSH_PATH_WORKDIR`.

### Environment-variable mapping

A config key maps to an env var by: prefix with `CHEATSH_`, replace `.` with `_`,
uppercase. Only **scalar** (int/string) values can be set this way.

| Config key | Environment variable |
|------------|----------------------|
| `server.port` | `CHEATSH_SERVER_PORT` |
| `server.bind` | `CHEATSH_SERVER_BIND` |
| `cache.type` | `CHEATSH_CACHE_TYPE` |
| `cache.redis.host` | `CHEATSH_CACHE_REDIS_HOST` |
| `cache.redis.port` | `CHEATSH_CACHE_REDIS_PORT` |
| `cache.redis.db` | `CHEATSH_CACHE_REDIS_DB` |
| `cache.redis.prefix` | `CHEATSH_CACHE_REDIS_PREFIX` |
| `log.level` | `CHEATSH_LOG_LEVEL` |
| `upstream.url` | `CHEATSH_UPSTREAM_URL` |
| `upstream.timeout` | `CHEATSH_UPSTREAM_TIMEOUT` |
| `search.limit` | `CHEATSH_SEARCH_LIMIT` |

> [!CAUTION]
> Two easy mistakes:
> 1. **`CHEATSH_PORT` is special-cased in `bin/srv.py`** and read directly — it is
>    *not* the same path as the config mapping. It takes precedence over
>    `server.port` when set. `CHEATSH_SERVER_PORT` works too, via config.
> 2. The env var must match the **full** dotted key. `CHEATSH_REDIS_PREFIX` (seen
>    in `tests/run-tests.sh`) does **not** map to anything — the correct variable
>    is `CHEATSH_CACHE_REDIS_PREFIX`.

### Full default configuration

| Key | Default | Meaning |
|-----|---------|---------|
| `adapters.active` | tldr, cheat, fosdem, translation, rosetta, late.nz, question, cheat.sheets, cheat.sheets dir, learnxiny, rfc, oeis, chmod | Enabled sources |
| `adapters.mandatory` | `["search"]` | Always-on adapters |
| `cache.type` | `redis` | `redis` or `none` |
| `cache.redis.host` | `localhost` | Redis host |
| `cache.redis.port` | `6379` | Redis port |
| `cache.redis.db` | `0` | Redis database number |
| `cache.redis.prefix` | `""` | Optional key prefix |
| `frontend.styles` | (all Pygments styles) | Allowed `?style=` values |
| `log.level` | `4` | Verbosity (≥5 prints extra adapter debug) |
| `path.repositories` | `upstream` | Where upstream repos are cloned |
| `path.spool` | `spool` | Where POSTed cheat sheets are stored |
| `path.workdir` | `$HOME/.cheat.sh` | Base work dir for all relative paths |
| `path.log.main` | `log/main.log` | Application log |
| `path.log.queries` | `log/queries.log` | Per-query log |
| `path.log.fetch` | `log/fetch.log` | Fetch/update log |
| `path.internal.*` | (derived from project dir) | Static assets, pages, templates, `ansi2html.sh`, upstream binary |
| `routing.pre` / `routing.main` / `routing.post` / `routing.default` | (see [§5](#5-routing-engine)) | Routing table |
| `upstream.url` | `https://cheat.sh` | Fallback/proxy server |
| `upstream.timeout` | `5` | Upstream request timeout (seconds) |
| `search.limit` | `20` | Max answers returned for a search |
| `server.bind` | `0.0.0.0` | Bind address |
| `server.port` | `8002` | Listen port |

### `etc/config.yaml` (shipped example)

```yaml
server:
    address: "0.0.0.0"
cache:
    type:   redis
```

> [!WARNING]
> The shipped `etc/config.yaml` uses the key `server.address`, but the real
> configuration key is **`server.bind`**. `server.address` does not map to any
> known key and is silently ignored (the default `0.0.0.0` is used regardless).
> When editing config, use the keys from the table above, not the example file's
> naming.

### Request (per-query) options

Parsed by `lib/options.py` from the query string:

| Letter | Effect (internal flag) |
|--------|------------------------|
| `c` | `add_comments=False`, `unindent_code=False` |
| `C` | `add_comments=False`, `unindent_code=True` |
| `Q` | `remove_text=True` (code only) |
| `q` | `quiet=True` (suppress HTML chrome) |
| `T` | `no-terminal=True` (strip ANSI colors) |

Defaults: `add_comments=True`. Key-value args like `style=native` and `lang=de`
pass through directly.

---

## 8. Deployment

### Docker Compose (recommended)

The simplest self-hosted setup uses the bundled `docker-compose.yml`:

```bash
docker-compose up
```

This builds the app image, fetches every cheat sheet into the image at build time,
and starts two services:

```yaml
services:
  app:
    build: .
    image: cheat.sh
    container_name: chtsh
    depends_on:
      - redis
    environment:
      - CHEATSH_CACHE_REDIS_HOST=redis
    ports:
      - "8002:8002"
    volumes:
      - .:/app:Z
  redis:
    image: redis:4-alpine
    volumes:
      - redis_data:/data
```

The service is then available at `http://localhost:8002`.

> [!IMPORTANT]
> The README explicitly notes the Docker setup is an early implementation,
> recommended for internal/dev/personal use rather than large public production.

### The image build (`Dockerfile`)

Key build stages (Alpine 3.14 base):

1. Install runtime packages: `git`, `py3-six`, `py3-pygments`, `py3-yaml`,
   `py3-gevent`, `libstdc++`, `py3-colorama`, `py3-requests`, `py3-icu`,
   `py3-redis`, `sed`.
2. Copy the source into `/app`.
3. Install remaining Python deps via a **temporary** build toolchain
   (`py3-pip`, `g++`, `python3-dev`, `libffi-dev`), then remove it.
4. **Fetch all cheat sheets** at build time:
   `python3 lib/fetch.py fetch-all` (into `/root/.cheat.sh/`).
5. Install server packages: `py3-jinja2`, `py3-flask`, `bash`, `gawk`.
6. Entry point: `python3 -u bin/srv.py`.

### Debug deployment

`docker-compose.debug.yml` overrides the app to run the Flask dev server (with
autoreload and tracebacks) instead of gevent:

```bash
docker-compose -f docker-compose.yml -f docker-compose.debug.yml up
```

It sets `FLASK_ENV=development`, `FLASK_APP=bin/app.py`, `FLASK_RUN_HOST=0.0.0.0`,
`FLASK_RUN_PORT=8002`.

> [!CAUTION]
> Never expose the Flask dev server (`docker-compose.debug.yml`) to the public
> internet. It is single-threaded, not hardened, and prints tracebacks. Use
> `bin/srv.py` (gevent) for anything reachable.

### Running without Docker

```bash
# 1. Install dependencies
pip3 install -r requirements.txt

# 2. Fetch all upstream cheat sheets
python3 lib/fetch.py fetch-all

# 3. Start the production server (gevent)
python3 bin/srv.py                 # binds server.bind:server.port (0.0.0.0:8002)
CHEATSH_PORT=8080 python3 bin/srv.py   # override the port
python3 bin/srv.py --debug         # enable Flask debug (limited under gevent)
```

A reverse proxy (nginx, Traefik, …) typically terminates TLS in front of
`bin/srv.py`. The app reads `X-Forwarded-For` for the client IP, so configure your
proxy to set it.

---

## 9. Operations

### Fetching and updating cheat sheets

All cheat-sheet content lives in cloned upstream repos under
`path.repositories` (`upstream/` in the work dir). Manage them with `lib/fetch.py`:

```bash
python3 lib/fetch.py fetch-all     # initial clone of every configured repo
python3 lib/fetch.py update-all    # git pull each repo + invalidate changed cache entries
python3 lib/fetch.py update NAME   # update one adapter's repo
```

`update-all` is the routine maintenance command: for each git-backed adapter it
pulls, computes the changed files since the last recorded state
(`.cached_revision`), and **invalidates exactly those cache entries** so updated
sheets are re-rendered while everything else stays warm.

> [!TIP]
> Schedule `python3 lib/fetch.py update-all` (e.g. via cron) to keep content fresh.
> Because it invalidates only changed entries, it is cheap to run often.

### Logs

| Log | Path (relative to work dir) | Contents |
|-----|------------------------------|----------|
| Main | `log/main.log` | Application/Flask logging (DEBUG level) |
| Queries | `log/queries.log` | One line per query: `IP found topic user_agent` |
| Fetch | `log/fetch.log` | Repository fetch/update activity |

Flask exceptions are additionally sent to `stderr` (with werkzeug duplicate-line
filtering), so container logs surface errors.

### Cache maintenance

```bash
# delete specific Redis keys directly
python3 bin/clean_cache.py "tldr:tar" "cheat.sheets:python/lambda"
```

To flush everything, use `redis-cli FLUSHDB` against the configured database (be
aware this also drops session/last-query keys).

### Submitting cheat sheets (spool)

POSTed cheat sheets (`:post`) are written to `path.spool` as
`topic.name.<nonce>` files for later review/merge. Monitor and drain this directory
as part of operations.

---

## 10. Caching

Caching is implemented in `lib/cache.py` and controlled per-adapter via
`is_cache_needed()`.

### Key scheme

| Key pattern | Used for |
|-------------|----------|
| `<topic_type>:<topic>` | Standard cached answers (e.g. `tldr:tar`) |
| `q:<topic>` | StackOverflow question answers (legacy, expensive) |
| `l:<client_id>` | A client's last query (for `:last`) |

An optional global prefix (`cache.redis.prefix`) is prepended as `<prefix>:`.

### Behavior

- Values that are `dict`/`list` are JSON-serialized on `put` and parsed on `get`.
- Adapters that set `_cache_needed = True` (tldr, cheat, learnxiny, rosetta,
  translation, rfc, oeis, chmod, question) are cached; others (cheat.sheets,
  internal, search, upstream failures) are not.
- An adapter's `_get_page()` may override caching per-answer by returning a dict
  with a `"cache"` key (the `UpstreamAdapter` marks offline answers
  `"cache": False`).

> [!NOTE]
> Switching `cache.type` to `none` (as the standalone CLI and the test harness do)
> disables Redis entirely — every request is computed fresh. Useful for testing or
> a no-Redis deployment, at the cost of latency for expensive sources.

---

## 11. Security

cheat.sh has a deliberately small attack surface (no auth, no DB, no user data),
but operators should understand the following.

### Input handling

- **Query sanitization** — `cheat_wrapper._sanitize_query()` strips `< > "` from
  the query before processing, reducing reflected-content risk in the HTML page.
- **Command adapters run shell programs.** `CommandAdapter`, `AdapterRfc`,
  `AdapterOeis`, and `AdapterChmod` execute helper scripts via `subprocess` with
  **argument lists** (not shell strings), and the chmod/oeis adapters scrub their
  topic input with regexes (`[^a-zA-Z0-9-]` etc.) before passing it. This avoids
  shell injection through the topic.
- **The `fosdem` and `translation` adapters invoke fixed binaries.** `fosdem` uses
  a `sudo` NOPASSWD entry for a single demo command, and `translation` references
  a hard-coded path (`/home/igor/cheat.sh/bin/get_translation`). These are
  environment-specific and inert on a default self-host.

### Rate limiting

`lib/limits.py` caps queries **per source IP** for multi-word queries (those
containing `+`):

| Interval | Limit |
|----------|-------|
| Minute | 30 |
| Hour | 600 |
| Day | 1000 |

Over-limit requests receive `429 Not so fast!`. There is a hard-coded whitelist
(`_WHITELIST = ["5.9.243.177"]`). The client IP is taken from `X-Forwarded-For`
(first value) when present, else `remote_addr`.

> [!WARNING]
> Because the IP comes from `X-Forwarded-For`, **only trust it behind a proxy you
> control**. If `bin/srv.py` is directly internet-facing, a client can spoof the
> header and bypass rate limits. Always front the service with a reverse proxy that
> sets `X-Forwarded-For` from the real connection.

### The `:shell-x` proxy

`bin/app.py` contains a `_proxy()` path for `/:shell-x/` that forwards to a local
service on port `3000`. This is a special/experimental feature and a known source
of `500` errors (see issues). If you do not run that backend, treat `:shell-x`
paths as unsupported.

> [!CAUTION]
> Review the `_proxy()` code before exposing it. It forwards method, headers
> (minus `Host`), data, and cookies to a constructed URL. Leaving it reachable
> without the intended backend produces errors; misconfiguring the target could
> create an SSRF-style forwarder. Disable or guard it for public deployments.

### Hardening checklist

- [ ] Terminate TLS at a reverse proxy; set `X-Forwarded-For` from the real peer.
- [ ] Run `bin/srv.py` (gevent), never the Flask dev server, in production.
- [ ] Run the container as a non-root user where possible (the stock Dockerfile
      fetches into `/root/.cheat.sh`).
- [ ] Pin and regularly rebuild the image (Alpine 3.14 and `redis:4-alpine` are
      old — see [§17](#17-potential-improvements)).
- [ ] Disable or firewall the `:shell-x` proxy unless intentionally used.
- [ ] Monitor `log/queries.log` for abuse patterns.

---

## 12. Repository Structure

```text
cheat.sh/
├── bin/
│   ├── app.py            Flask application (routes, UA detection, limits, POST)
│   ├── srv.py            gevent production server (entry point)
│   ├── clean_cache.py    delete specific Redis keys
│   └── release.py        release helper
├── lib/
│   ├── cheat_wrapper.py  query parse → route → postprocess → visualize
│   ├── routing.py        Router + routing table
│   ├── config.py         layered configuration (CONFIG dict)
│   ├── cache.py          Redis / no-op cache
│   ├── fetch.py          clone/update upstream repos + cache invalidation
│   ├── limits.py         per-IP rate limiting
│   ├── options.py        request-option parsing
│   ├── post.py           cheat-sheet submission (spool)
│   ├── postprocessing.py comment-adding + keyword filtering
│   ├── search.py         keyword search across topics
│   ├── standalone.py     offline CLI entry point
│   ├── stateful_queries.py  per-client last-query storage
│   ├── languages_data.py    language aliases, lexers, editor filetype maps
│   ├── globals.py        fatal/error/log helpers
│   ├── buttons.py        GitHub/Twitter HTML buttons
│   ├── adapter/          all source adapters (see §4)
│   ├── fmt/              comments / internal / markdown formatting
│   ├── frontend/         ansi.py, html.py renderers
│   └── panela/           internal coloring library
├── share/
│   ├── cht.sh.txt        the cht.sh client (served at /:cht.sh)
│   ├── ansi2html.sh      ANSI→HTML converter (used by frontend/html.py)
│   ├── bash_completion.txt, zsh.txt, fish.txt   shell completions
│   ├── vim.txt, emacs.txt, emacs-ivy.txt        editor helpers
│   ├── help.txt, intro.txt, firstpage*.txt      internal pages
│   ├── adapters/         chmod.sh, oeis.sh, rfc.sh (+ chmod.grc)
│   └── static/           CSS, images, opensearch.xml, malformed-response.html
├── etc/config.yaml       example config (note server.address quirk)
├── tests/                run-tests.sh, tests.txt, results/, unit tests
├── doc/                  standalone.md, README-ja.md
├── Dockerfile
├── docker-compose.yml
├── docker-compose.debug.yml
└── requirements.txt
```

> [!NOTE]
> Internal pages such as `:help`, `:intro`, `:bash_completion`, and `:vim` are
> served verbatim from the corresponding `share/*.txt` files by the
> `InternalPages` adapter. To change `:help`, edit `share/help.txt`.

---

## 13. Module Reference

### `bin/app.py`

The Flask application. Routes:

| Route | Methods | Purpose |
|-------|---------|---------|
| `/` and `/<path:topic>` | GET, POST | Main query handler (`answer`) |
| `/files/<path:path>` | GET | Serve static assets from `path.internal.static` |
| `/favicon.ico` | GET | Serve favicon |
| `/malformed-response.html` | GET | Serve the malformed-response page |

Helpers: `_is_html_needed()`, `get_request_ip()`, `get_answer_language()`,
`log_query()`, `is_result_a_script()`, `_proxy()`.

### `lib/cheat_wrapper.py`

`cheat_wrapper(query, request_options, output_format)` — the single function the
app calls. Normalizes the query (sanitize, add section name, rewrite aliases and
editor section names), parses `~` searches, dispatches to `routing.get_answers()`
or `search.find_answers_by_keyword()`, postprocesses, and visualizes.

### `lib/routing.py`

`Router` builds the live adapter set from `adapters.active ∪ adapters.mandatory`,
assembles the routing table, and exposes `get_answers()`, `get_topic_type()`,
`get_topics_list()`, and `:random` handling. Topic-type results are memoized.

### `lib/config.py`

Builds `CONFIG` (a `Config(dict)` subclass that absolutizes `path.*` keys against
`path.workdir`). Loads defaults, then project file, then work-dir file, then env
vars. Includes a doctested `_get_nested()` for hierarchical YAML keys.

### `lib/fetch.py`

`fetch_all()` / `update_all()` / `update_by_name()` plus a CLI
(`fetch-all`, `update-all`, `update NAME`). Detects duplicate repository locations,
creates parent dirs, runs adapter-provided git commands, and invalidates cache
entries for changed files.

### `lib/frontend/ansi.py`

`visualize()` renders answer dicts to ANSI. `_colorize_ansi_answer()` selects a
Pygments lexer by language (`languages_data.LEXER`), applies a 256-color terminal
formatter, and optionally highlights only code blocks. `remove_ansi()` strips color
when `no-terminal` (`?T`) is set.

### `lib/frontend/html.py`

`visualize()` first renders ANSI, then pipes it through `share/ansi2html.sh`
(`bash` + `gawk` + `sed`) to produce HTML, injecting a search form, topic datalist,
edit button (for editable `cheat.sheets` topics), and GitHub/Twitter buttons.

### `lib/languages_data.py`

The canonical data tables: `LEXER` (language → Pygments lexer), `LANGUAGE_ALIAS`
(e.g. `c++`→`cpp`), `VIM_NAME`/`SO_NAME`/`EMACS_FT_NAME`/… (editor and StackOverflow
name mappings), and `rewrite_editor_section_name()` / `get_lexer_name()`.

### `lib/standalone.py`

Offline entry point: `lib/standalone.py [OPTIONS] QUERY`. Forces `cache.type=none`,
parses the query like a URL, and calls `cheat_wrapper()` directly — no server, no
Redis. This is what the test harness uses in standalone mode.

---

## 14. Code Patterns & Conventions

- **Adapter polymorphism via subclass discovery.** New sources require no
  registry edit beyond config — `all_adapters()` finds every `Adapter` subclass at
  runtime. This is the project's defining extensibility pattern.
- **Configuration as a flat dotted-key dict.** `CONFIG["cache.redis.host"]` rather
  than nested access; `Config.__setitem__` transparently absolutizes path keys.
- **Answer dicts as the internal contract.** Adapters return
  `{topic, topic_type, format, cache, answer[, filetype]}`; frontends consume that
  shape. `format` is one of `code`, `text`, `text+code`, `ansi`.
- **ANSI-first rendering, HTML as a transform.** HTML is produced by converting
  ANSI output, so terminal and browser views stay consistent.
- **Graceful degradation.** Missing upstream binary → proxy to public server;
  connection failure → "Are you offline?" banner; unknown topic → fuzzy suggestions
  instead of a hard error.
- **Python 2/3 compatibility shims** remain in several modules
  (`from __future__ import print_function`, `six.with_metaclass`, `urlparse`
  fallbacks) though the project targets Python 3.

> [!NOTE]
> `lib/search.py` is explicitly described in its own docstring as "very naive…
> just a placeholder" — it iterates topics and pattern-matches. It is correct but
> not optimized; large recursive searches are bounded by `search.limit`.

---

## 15. Testing

The test suite lives in `tests/`:

```bash
# Unit tests (pytest)
python3 -m pytest -v lib/

# Input/output tests (golden-file comparison)
cd tests && ./run-tests.sh
```

`run-tests.sh` reads `tests/tests.txt` (one query per line) and diffs the output
against recorded golden files in `tests/results/`. It can run in three modes:

- **standalone** (default) — invokes `lib/standalone.py` (no server needed).
- **client** — runs queries through `share/cht.sh.txt` against a live server.
- **curl** — hits a running server at `CHTSH_URL` (default `http://localhost:8002`).

Useful environment toggles for the harness:

| Variable | Default | Effect |
|----------|---------|--------|
| `CHEATSH_TEST_STANDALONE` | `YES` | Use standalone mode |
| `CHEATSH_TEST_SKIP_ONLINE` | `NO` | Skip tests tagged `[online]` |
| `CHEATSH_UPDATE_TESTS_RESULTS` | `NO` | Re-record golden files instead of failing |
| `CHTSH_URL` | `http://localhost:8002` | Server for client/curl modes |

To run a server for the curl/client modes:

```bash
CHEATSH_CACHE_TYPE=none CHEATSH_PORT=50000 python3 bin/srv.py   # no caching (recommended)
```

There is also a focused unit test, `lib/cheat_wrapper_test.py`, that verifies the
`_add_section_name()` query-rewriting logic (e.g. `g++` is not split, but
`python copy file` becomes `python/copy file`).

---

## 16. Troubleshooting (Operator)

### `500 Internal Server Error` on normal queries

**Cause:** An adapter raised, an upstream timed out, or the `:shell-x` proxy was
hit without its backend. **Solution:** Check `log/main.log` and container `stderr`;
identify the failing adapter; ensure upstream repos are fetched
(`lib/fetch.py fetch-all`) and Redis is reachable. **Prevention:** Health-check
Redis and the upstream binary; disable `:shell-x` if unused.

### `docker-compose up` fails to build or start

**Cause:** Native build failures (ICU/`polyglot`), or an outdated base image
pulling incompatible package versions (e.g. a werkzeug/flask mismatch surfacing as
`cannot import name '_plain_int' from 'werkzeug._internal'`). **Solution:** Pin
compatible Flask/werkzeug versions; rebuild with `--no-cache`. **Prevention:**
Modernize the base image and pin dependency versions (see [§17](#17-potential-improvements)).

### Answers never update

**Cause:** Cache entries aren't being invalidated; upstream repos aren't being
pulled. **Solution:** Run `python3 lib/fetch.py update-all`; verify
`.cached_revision` files exist in each `upstream/<repo>` checkout. **Prevention:**
Cron `update-all` regularly.

### Rate limiting blocks legitimate traffic (or fails to block abuse)

**Cause:** `X-Forwarded-For` is missing (everyone shares one apparent IP) or
spoofable (no trusted proxy). **Solution:** Put a reverse proxy in front that sets
`X-Forwarded-For` from the real connection. **Prevention:** Never expose
`bin/srv.py` directly.

### HTML output is empty or errors

**Cause:** `ansi2html.sh` needs `bash` and `gawk`; if missing, HTML rendering
fails. **Solution:** Ensure `bash` and `gawk` are installed in the runtime image
(the Dockerfile installs them in the final stage). **Prevention:** Keep the HTML
toolchain in your image.

### `:last` returns an error

**Cause:** `:last` requires a session-id cookie. **Solution:** Have clients set an
`id` cookie (the `cht.sh` client does this via its `id` command). This is expected
behavior, not a fault.

---

## 17. Potential Improvements

Source-derived observations for maintainers and operators:

- **Modernize the container base.** `Dockerfile` pins Alpine 3.14 and
  `docker-compose.yml` uses `redis:4-alpine` — both are old. Upgrading reduces CVE
  exposure and avoids the Flask/werkzeug import incompatibilities reported in
  issues.
- **Pin Python dependency versions.** `requirements.txt` is largely unpinned
  (only `colored<1.4.3`). Pinning would prevent the `werkzeug._internal._plain_int`
  class of breakages.
- **Fix the `etc/config.yaml` example.** It uses `server.address`, which is not a
  real config key; it should be `server.bind` to actually take effect.
- **Reconcile the port env var.** `bin/srv.py` reads `CHEATSH_PORT` while the
  config system expects `CHEATSH_SERVER_PORT`. Documenting or unifying these would
  avoid confusion.
- **Replace the placeholder search.** `lib/search.py` self-describes as naive; an
  indexed/ranked search would improve `~keyword` quality and remove the hard
  `search.limit` cliff.
- **Harden or remove `:shell-x`.** The `_proxy()` path is experimental, a frequent
  `500` source, and a potential SSRF vector if misconfigured. Gate it behind config
  or remove it from default builds.
- **Make rate-limit state shareable.** `Limits` keeps counters in process memory,
  so multiple gevent workers/instances don't share limits. Backing them with Redis
  would make limiting consistent across a scaled-out deployment.
- **Document `CHEATSH_CACHE_REDIS_PREFIX`.** The test script's `CHEATSH_REDIS_PREFIX`
  is a no-op; correcting it prevents accidental cache-namespace collisions in CI.
- **Run the container as non-root.** Cheat sheets are fetched into `/root/.cheat.sh`;
  a dedicated unprivileged user would tighten the security posture.

### Known issues (community-reported)

| Issue | Summary | Operator note |
|-------|---------|---------------|
| #423, #381, #383, #422 | `500 Internal Server Error` | Often `:shell-x` or upstream timeouts |
| #406, #427 | `docker-compose up` import/build errors | werkzeug/flask version drift |
| #390 | `pip install` on Windows fails | Client is a shell script, not a pip package |
| #430 | Install hangs on macOS | Network stall / missing `rlwrap` |
| #403 | History file location via env var (feature) | Client enhancement |
| #433, #432, #425 | Light-theme readability / source content | UI + upstream content |

---

## 18. Quick Reference

### Entry points

| Command | Purpose |
|---------|---------|
| `python3 bin/srv.py` | Production gevent server (`server.bind:server.port`) |
| `python3 bin/srv.py --debug` | Enable Flask debug (limited under gevent) |
| `CHEATSH_PORT=N python3 bin/srv.py` | Override the listen port |
| `python3 lib/standalone.py QUERY` | Offline, server-less query |
| `python3 lib/fetch.py fetch-all` | Clone all upstream cheat-sheet repos |
| `python3 lib/fetch.py update-all` | Update repos + invalidate changed cache |
| `python3 bin/clean_cache.py KEY...` | Delete specific Redis keys |

### Key environment variables

| Variable | Effect |
|----------|--------|
| `CHEATSH_PORT` | Listen port (read directly by `srv.py`, highest precedence) |
| `CHEATSH_CACHE_TYPE` | `redis` or `none` |
| `CHEATSH_CACHE_REDIS_HOST` / `_PORT` / `_DB` / `_PREFIX` | Redis connection |
| `CHEATSH_PATH_WORKDIR` | Override the work dir (default `$HOME/.cheat.sh`) |
| `CHEATSH_CONFIG` | Override the config-file path |
| `CHEATSH_UPSTREAM_URL` / `_TIMEOUT` | Fallback/proxy server |
| `CHEATSH_SEARCH_LIMIT` | Max search results |

### Default ports & paths

| Item | Value |
|------|-------|
| Listen | `0.0.0.0:8002` |
| Redis | `localhost:6379`, db `0` |
| Work dir | `$HOME/.cheat.sh` |
| Upstream repos | `<workdir>/upstream/` |
| Spool (POSTed sheets) | `<workdir>/spool/` |
| Logs | `<workdir>/log/{main,queries,fetch}.log` |

### Active adapters (default)

`tldr`, `cheat`, `fosdem`, `translation`, `rosetta`, `late.nz`, `question`,
`cheat.sheets`, `cheat.sheets dir`, `learnxiny`, `rfc`, `oeis`, `chmod`
(+ mandatory `search`, + always-on `internal`/`unknown`).

---

_Generated June 2026 from chubin/cheat.sh at 031a5d3. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._